Congestion control  in storage systems

ABSTRACT

An I/O request directed to a portion of a storage object managed at a distributed storage service is received. A congestion control parameter value to be used to schedule a storage operation corresponding to the I/O request is determined. The congestion control parameter is based at least in part on an offset within the storage object to which the I/O request is directed. The storage operation is scheduled in accordance with the congestion control parameter at a selected physical storage device to which the portion of the storage object is mapped.

This application is a continuation of U.S. patent application Ser. No.14/231,063, filed Mar. 31, 2014, now U.S. Pat. No. 9,274,710, which ishereby incorporated by reference in its entirety.

BACKGROUND

Many companies and other organizations operate computer networks thatinterconnect numerous computing systems to support their operations,such as with the computing systems being co-located (e.g., as part of alocal network) or instead located in multiple distinct geographicallocations (e.g., connected via one or more private or publicintermediate networks). For example, data centers housing significantnumbers of interconnected computing systems have become commonplace,such as private data centers that are operated by and on behalf of asingle organization, and public data centers that are operated byentities as businesses to provide computing resources to customers. Somepublic data center operators provide network access, power, and secureinstallation facilities for hardware owned by various customers, whileother public data center operators provide “full service” facilitiesthat also include hardware resources made available for use by theircustomers.

Some large provider networks implement a variety of storage services,such as services that implement block-level devices (volumes) or objectsthat can be modeled as arbitrary bit buckets accessible via respectiveURLs. However, a number of applications running at data centers of aprovider network may still face limitations with respect to their use ofsome of the more common storage-related programmatic interfaces, such asvarious industry-standard file system interfaces. Some industry-standardfile systems may have been designed prior to the large-scale deploymentof network-accessible services, and may therefore support consistencymodels and other semantics that are not straightforward to implement indistributed systems in which asynchronous computational interactions,failures of individual components and network partitions ornetworking-related delays are all relatively common.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 provides a high-level overview of a distributed file storageservice, according to at least some embodiments.

FIG. 2 illustrates the use of resources at a plurality of availabilitycontainers of a provider network to implement a file storage service,according to at least some embodiments.

FIG. 3 illustrates a configuration in which network addresses associatedwith isolated virtual networks are assigned to access subsystem nodes ofa storage service, according to at least some embodiments.

FIG. 4 illustrates a mapping between file storage service objects,logical blocks, and physical pages at one or more extents, according toat least some embodiments.

FIG. 5 illustrates a configuration of replica groups for data andmetadata extents, according to at least some embodiments.

FIG. 6 illustrates examples of interactions associated with cachingmetadata at access subsystem nodes of a file storage service, accordingto at least some embodiments.

FIG. 7 illustrates examples of the use of distinct combinations ofpolicies pertaining to data durability, performance, andlogical-to-physical data mappings for file stores, according to at leastsome embodiments.

FIG. 8a is a flow diagram illustrating aspects of configuration andadministration-related operations that may be performed to implement ascalable distributed file system storage service, according to at leastsome embodiments.

FIG. 8b is a flow diagram illustrating aspects of operations that may beperformed in response to client requests at a scalable distributed filesystem storage service, according to at least some embodiments.

FIG. 9 is a flow diagram illustrating aspects of operations that may beperformed to implement a replication-based durability policy at adistributed file system storage service, according to at least someembodiments.

FIG. 10 is a flow diagram illustrating aspects of operations that may beperformed to cache metadata at an access subsystem node of a distributedfile system storage service, according to at least some embodiments.

FIG. 11 illustrates examples of read-modify-write sequences that may beimplemented at a file storage service in which write offsets and writesizes may sometimes not be aligned with the boundaries of atomic unitsof physical storage, according to at least some embodiments.

FIG. 12 illustrates the use of consensus-based replicated state machinesfor extent replica groups, according to at least some embodiments.

FIG. 13 illustrates example interactions involved in a conditional writeprotocol that may be used for some types of write operations, accordingto at least some embodiments.

FIG. 14 illustrates example write log buffers that may be established toimplement a conditional write protocol, according to at least someembodiments.

FIG. 15 is a flow diagram illustrating aspects of operations that may beperformed to implement a conditional write protocol at a distributedfile system storage service, according to at least some embodiments.

FIG. 16 illustrates an example message flow that may result in a commitof a distributed transaction at a file storage service, according to atleast some embodiments.

FIG. 17 illustrates an example message flow that may result in an abortof a distributed transaction at a file storage service, according to atleast some embodiments.

FIG. 18 illustrates an example of a distributed transaction participantnode chain that includes a node designated as the coordinator of thetransaction, according to at least some embodiments.

FIG. 19 illustrates example operations that may be performed tofacilitate distributed transaction completion in the event of a failureat one of the nodes of a node chain, according to at least someembodiments.

FIG. 20 is a flow diagram illustrating aspects of operations that may beperformed to coordinate a distributed transaction at a file systemstorage service, according to at least some embodiments.

FIG. 21 is a flow diagram illustrating aspects of operations that may beperformed in response to receiving a transaction-prepare message at anode of a storage service, according to at least some embodiments.

FIG. 22 is a flow diagram illustrating aspects of operations that may beperformed in response to receiving a transaction-commit message at anode of a storage service, according to at least some embodiments.

FIG. 23 is a flow diagram illustrating aspects of operations that may beperformed in response to receiving a transaction-abort message at a nodeof a storage service, according to at least some embodiments.

FIG. 24 illustrates examples of over-subscribed storage extents at adistributed storage service, according to at least some embodiments.

FIG. 25 illustrates interactions among subsystems of a storage serviceimplementing on-demand physical page-level allocation and extentoversubscription, according to at least some embodiments.

FIG. 26a illustrates an extent for which a free space threshold has beendesignated, while FIG. 26b illustrates an expansion of the extentresulting from a violation of the free space threshold, according to atleast some embodiments.

FIG. 27 is a flow diagram illustrating aspects of operations that may beperformed to implement on-demand physical page allocation at extentsthat support oversubscription, according to at least some embodiments.

FIG. 28 is a flow diagram illustrating aspects of operations that may beperformed to dynamically modify extent oversubscription parameters,according to at least some embodiments.

FIG. 29 illustrates examples of file store objects striped usingvariable stripe sizes, according to at least some embodiments.

FIG. 30 illustrates examples of stripe sizing sequences that may be usedfor file store objects, according to at least some embodiments.

FIG. 31 illustrates examples of factors that may be taken intoconsideration at a metadata subsystem to make stripe sizing and/orconsolidation decisions for file store objects, according to at leastsome embodiments.

FIG. 32 is a flow diagram illustrating aspects of operations that may beperformed to implement striping using variable stripe sizes, accordingto at least some embodiments.

FIG. 33 illustrates an example timeline of the progress made by multipleconcurrent read requests directed to a logical block of a storageservice object in a scheduling environment in which all the readrequests to the logical block are granted equal priority relative to oneanother, according to at least some embodiments.

FIG. 34 illustrates an example timeline of the progress made by multipleconcurrent read requests directed to a logical block of a storageservice object in a scheduling environment in which an offset-basedcongestion control policy is used, according to at least someembodiments.

FIG. 35a illustrates an example of a token-based congestion controlmechanism that may be used for scheduling I/O requests at a storageservice, wile FIG. 35b illustrates examples of offset-based tokenconsumption policies that may be employed, according to at least someembodiments.

FIG. 36 illustrates an example of the use of offset-based delays forcongestion control at a storage service, according to at least someembodiments.

FIG. 37 illustrates examples of congestion control policies that may bedependent on the type of storage object being accessed and variouscharacteristics of the requested accesses, according to at least someembodiments.

FIG. 38 is a flow diagram illustrating aspects of operations that may beperformed to implement offset-based congestion control for schedulingI/O operations at a storage service, according to at least someembodiments.

FIG. 39 illustrates an example of the metadata changes that may have tobe performed at a plurality of metadata subsystem nodes of a storageservice to implement a rename operation, according to at least someembodiments.

FIG. 40 illustrates a use of a deadlock avoidance mechanism forconcurrent rename operations, according to at least some embodiments.

FIG. 41 is a flow diagram illustrating aspects of operations that may beperformed to implement a first rename workflow based on a first lockordering, among two possible lock orderings, that may be determined at astorage service for a rename operation, according to at least someembodiments.

FIG. 42 is a flow diagram illustrating aspects of operations that may beperformed to implement a second rename workflow based on a second lockordering, among the two possible lock orderings, that may be determinedat a storage service for a rename operation, according to at least someembodiments.

FIG. 43 is a flow diagram illustrating aspects of recovery operationsthat may be performed in response to a failure of one metadata subsystemnode of a pair of metadata subsystem nodes participating in a renameworkflow, according to at least some embodiments.

FIG. 44 is a flow diagram illustrating aspects of recovery operationsthat may be performed in response to a failure of the other metadatasubsystem node of the pair of metadata subsystem nodes participating inthe rename workflow, according to at least some embodiments.

FIG. 45 illustrates an example of a hash-based directed acyclic graph(DAG) that may be used for file store namespace management, according toat least some embodiments.

FIG. 46 illustrates a technique for traversing an HDAG using successivesubsequences of a hash value obtained for a file name, according to atleast some embodiments.

FIG. 47 illustrates an example of the first of two types of HDAG nodesplits that may result from an attempt to insert an entry into anamespace, according to at least some embodiments.

FIG. 48 illustrates an example of the second of two types of HDAG nodesplits that may result from an attempt to insert an entry into anamespace, according to at least some embodiments.

FIG. 49 illustrates an example of the first of two types of HDAG nodedeletion operations, according to at least some embodiments.

FIG. 50 illustrates an example of the second of two types of HDAG nodedeletion operations, according to at least some embodiments.

FIG. 51 is a flow diagram illustrating aspects of operations that may beperformed in response to an insertion of an entry into a namespace thatresults in a first type of HDAG node split, according to at least someembodiments.

FIG. 52 is a flow diagram illustrating aspects of operations that may beperformed in response to an insertion of an entry into a namespace thatresults in a second type of HDAG node split, according to at least someembodiments.

FIG. 53 is a flow diagram illustrating aspects of operations that may beperformed in response to a deletion of an entry from a namespace,according to at least some embodiments.

FIG. 54 illustrates two dimensions of metadata that may be maintainedfor session-oriented file system protocols at a distributed storageservice, according to at least some embodiments.

FIG. 55 illustrates an example of client session metadata-relatedinteractions between subcomponents of a distributed storage service,according to at least some embodiments.

FIG. 56 illustrates alternative approaches to client session leaserenewal at a distributed storage service, according to at least someembodiments.

FIGS. 57a and 57b illustrate alternative approaches to lock statemanagement for a session-oriented file system protocol at a distributedstorage service, according to at least some embodiments.

FIG. 58 is a flow diagram illustrating aspects of client sessionmetadata management operations that may be performed a distributedstorage service, according to at least some embodiments.

FIG. 59 is a flow diagram illustrating aspects of client session leaserenewal operations that may be performed a distributed storage service,according to at least some embodiments.

FIG. 60 illustrates a system in which a load balancer layer isconfigured for a distributed storage service, according to at least someembodiments.

FIG. 61 illustrates example interactions between a load balancer nodeand a plurality of access subsystem nodes of a distributed storageservice, according to at least some embodiments.

FIG. 62 illustrates examples of connection acceptance criteria that mayvary with the number of connection attempts made, according to at leastsome embodiments.

FIG. 63 illustrates examples of connection acceptance criteria that maybe dependent on workload levels associated with a plurality ofresources, as well as on connection establishment attempt counts,according to at least some embodiments.

FIG. 64 is a flow diagram illustrating aspects of operations that may beperformed to implement connection balancing based on attempt counts at adistributed storage service, according to at least some embodiments.

FIG. 65 illustrates an example of an access subsystem of a distributedstorage service at which client connection re-balancing may be attemptedbased on workload indicators of members of a peer group of access nodes,according to at least some embodiments.

FIG. 66 illustrates an example of connection acceptance and re-balancingcriteria that may be used at an access subsystem node, according to atleast some embodiments.

FIG. 67 is a flow diagram illustrating aspects of operations that may beperformed at an access subsystem of a distributed storage service toimplement connection re-balancing, according to at least someembodiments.

FIG. 68 is a flow diagram illustrating aspects of operations that may beperformed at a distributed storage service to preserve client sessionsacross connection re-balancing events, according to at least someembodiments.

FIG. 69 is a block diagram illustrating an example computing device thatmay be used in at least some embodiments.

While embodiments are described herein by way of example for severalembodiments and illustrative drawings, those skilled in the art willrecognize that embodiments are not limited to the embodiments ordrawings described. It should be understood, that the drawings anddetailed description thereto are not intended to limit embodiments tothe particular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope as defined by the appended claims. The headings usedherein are for organizational purposes only and are not meant to be usedto limit the scope of the description or the claims. As used throughoutthis application, the word “may” is used in a permissive sense (i.e.,meaning having the potential to), rather than the mandatory sense (i.e.,meaning must). Similarly, the words “include,” “including,” and“includes” mean including, but not limited to.

DETAILED DESCRIPTION

Various embodiments of methods and apparatus for a high-availability,high-durability scalable file storage service are described. In at leastsome embodiments, the file storage service may be designed to supportshared access to files by thousands of clients, where each individualfile may comprise very large amounts (e.g., petabytes) of data, atperformance, availability and durability levels that are targeted to beindependent of the size of the file and/or the number of concurrentusers. One or more industry-standard file system interfaces or protocolsmay be supported by the service, such as various versions of NFS(network file system), SMB (Server Message Block), CIFS (Common InternetFile System) and the like. Accordingly, in at least some embodiments,the consistency models supported by the distributed file storage servicemay be at least as strong as the models supported by theindustry-standard protocols—for example, the service may supportsequential consistency. In a distributed system implementing asequential consistency model, the result of an execution of operationsimplemented collectively at a plurality of executing entities (e.g.,nodes or servers of the distributed system) is expected to be the sameas if all the operations were executed in some sequential order. Thefile storage service may be designed for use by a wide variety ofapplications, such as file content serving (e.g. web server farms,software development environments, and content management systems), highperformance computing (HPC) and “Big Data” applications such as media,financial, and scientific solutions requiring on-demand scaling of filestore capacity and performance, and the like. The term “file store” maybe used herein to indicate the logical equivalent of a file system—e.g.,a given client may create two different NFS-compliant file stores FS1and FS2, with the files of FS1 being stored within one set ofsubdirectories of a mountable root directory, and the files of FS2 beingstored within a set of subdirectories of a different mountable rootdirectory.

To help enable high levels of scalability, a modular architecture may beused for the service in at least some embodiments. For example, aphysical storage subsystem comprising some number of multi-tenantstorage nodes may be used for file store contents, while a logicallydistinct metadata subsystem with its own set of metadata nodes may beused for managing the file store contents in one implementation. Thelogical separation of metadata and data may be motivated, for example,by the fact that the performance, durability and/or availabilityrequirements for metadata may in at least some cases differ from (e.g.,more stringent than) the corresponding requirements for data. Afront-end access subsystem, with its own set of access nodes distinctfrom the metadata and storage nodes, may be responsible for exposingnetwork endpoints that allow clients to submit requests to create, read,update, modify and delete the file stores via the industry-standardinterfaces, and for handling connection management, load balancing,authentication, authorization and other tasks associated with clientinteractions. Resources may be deployed independently to any one of thesubsystems in some embodiments, e.g., to the access subsystem, themetadata subsystem, or the storage subsystem, without requiringcorresponding deployment changes at the other subsystems. For example,if a triggering condition such as a potential performance bottleneck isidentified in the access subsystem, or if some set of access subsystemnodes experience a network outage or other failure, additional accesssubsystem nodes may be brought online without affecting the storage ormetadata subsystems, and without pausing the flow of client requests.Similar deployment changes may be made at other subsystems as well inresponse to various types of triggering conditions. In some embodiments,the access subsystem nodes in particular may be implemented in a largelystateless manner, so that recovery from access node failures may beespecially efficient.

In at least some embodiments, the contents of the file store metadataobjects (e.g., data structures representing attributes of directoryentries, links, etc.) may themselves be stored on devices managed by thestorage subsystem—although, as described below, in some cases differentpolicies may be applied to the storage objects being used for the datathan are applied to the storage objects being used for metadata. In suchembodiments, the metadata subsystem nodes may, for example, comprisevarious processes or threads of execution that execute metadatamanagement logic and coordinate the storage of metadata contents at thestorage subsystem. A given storage subsystem node may include severaldifferent types of storage media in some embodiments, such as somenumber of devices employing rotating magnetic disks and some number ofdevices employing solid state drives (SSDs). In some embodiments a givenstorage subsystem node may store both metadata and data, either atrespective different storage devices or on the same storage device. Theterm “file store object” may be used herein to refer collectively todata objects such as files, directories and the like that are typicallyvisible to clients of the storage service, as well as to the internalmetadata structures (including for example the mappings between logicalblocks, physical pages and extents discussed below), used to manage andstore the data objects.

In at least some embodiments, the distributed file storage service maybe built using resources of a provider network, and may be designedprimarily to fulfill storage requests from other entities within theprovider network. Networks set up by an entity such as a company or apublic sector organization to provide one or more network-accessibleservices (such as various types of cloud-based computing or storageservices) accessible via the Internet and/or other networks to adistributed set of clients may be termed provider networks herein. Someof the services may be used to build higher-level services: for example,computing, storage or database services may be used as building blocksfor a content distribution service or a streaming data processingservice. At least some of the services of a provider network may bepackaged for client use in service units called “instances”: forexample, a virtual machine instantiated by a virtualized computingservice may represent a “compute instance”. Computing devices at whichsuch compute instances of the provider network are implemented may bereferred to herein as “instance hosts” or more simply as “hosts” herein.A given instance host may comprise several compute instances, and thecollection of compute instances at a particular instance host may beused to implement applications of one or more clients. In someembodiments, the file storage service may be accessible from some subset(or all) of the compute instances of a provider network, e.g., as aresult of assigning the appropriate network addresses to the accesssubsystem nodes of the storage service, implementing theauthorization/authentication protocols that are used for the virtualcomputing service, and so on. In some embodiments, clients outside theprovider network may also be provided access to the file storageservice. In various embodiments, at least some of the provider networkservices may implement a usage-based pricing policy—e.g., customers maybe charged for a compute instance based at least partly on how long theinstance was used, or on the number of requests of various types thatwere submitted from the compute instance. In at least some suchembodiments, the file storage service may also employ usage-basedpricing for at least some categories of client requests—e.g., theservice may keep records of the particular file system interfacerequests that were completed on behalf of a given customer, and maygenerate billing amounts for the customer on the basis of those records.

The file store service may support high levels of data durability insome embodiments, e.g., using any of a number of different replicationtechniques. For example, in one embodiment, file store data and metadatamay be physically stored using storage units called extents, and thecontents of an extent may be replicated at various physical storagedevices. The contents of an extent may be referred to herein as a“logical extent”, to distinguish it from the physical copies at thedifferent physical storage devices, which may be referred to as “extentreplicas”, “replica group members”, or “extentlets” or a “replicagroup”. In one implementation, for example, a file (or a metadataobject) may be organized as a sequence of logical blocks, with eachlogical block being mapped to one or more physical data pages. A logicalblock may considered a unit of striping, in that at least in someimplementations, the probability that the contents of two differentlogical blocks of the same file (or the same metadata structure) arestored at the same storage device may be low. Each replica of a givenlogical extent may comprise some number of physical data pages. In someembodiments, erasure-coding based extent replicas may be used, while inother embodiments, other replication techniques such as full replicationmay be used. In at least one embodiment, a combination of erasure codingand full replication may be used. A given modification request from aclient may accordingly be translated into a plurality of physicalmodifications at respective storage devices and/or respective storagesubsystem nodes, depending on the nature of the replication policy inuse for the corresponding file store object or metadata. In someembodiments, one or more of the extent replicas of a replica group maybe designated as a master replica, and updates to the extent may becoordinated, e.g., using a consensus-based replicated state machine, bythe storage service node that is hosting the current master. Such astorage service node may be termed a “master node” or a “leader” hereinwith respect to the extent for which it stores a master replica. In oneimplementation, if N extent replicas of a given logical extent are beingmaintained, a quorum of M (where M>=N/2) of the replicas may be needed,and such a quorum may be obtained using an update protocol initiated bythe leader/master node, before a particular update is committed. In oneembodiment, some extents may be used entirely for file contents or data,while other extents may be used exclusively for metadata. In otherembodiments, a given extent may store both data and metadata. In someimplementations, a consensus-based protocol may be used to replicate logrecords indicating state changes of a given file store, and the contentsof the state may be replicated using a plurality of extents (e.g., usingeither full replication or erasure-coded replicas). Replicated statemachines may also be used to ensure consistency for at least some typesof read operations in various embodiments. For example, a single clientread request may actually require a plurality of physical readoperations (e.g., of metadata and/or data) at various extents, and theuse of replicated state machines may ensure that the result of such adistributed read does not violate the read consistency requirements ofthe targeted file store.

A variety of different allocation and sizing policies may be used todetermine the sizes of, and relationships among, logical blocks,physical pages, and/or the extents for data and metadata in differentembodiments as described below. For example, in one straightforwardimplementation, a file may comprise some number of fixed size (e.g.,4-megabyte) logical blocks, each logical block may comprise some numberof fixed size (e.g., 32-kilobyte) physical pages, and each extent maycomprise sufficient storage space (e.g., 16 gigabytes) to store a fixednumber of pages. In other embodiments, different logical blocks maydiffer in size, physical pages may differ in size, or extents may differin size. Extents may be dynamically resized (e.g., grown or shrunk) insome embodiments. Static allocation may be used for logical blocks insome embodiments (e.g., all the physical storage for the entire logicalblock may be allocated in response to the first write directed to theblock, regardless of the size of the write payload relative to the sizeof the block), while dynamic allocation may be used in others. Varioustechniques and policies governing logical block configurations andcorresponding physical storage space allocations are described below infurther detail. In some embodiments, different file stores managed bythe file storage service may implement distinct block/page/extent sizingand configuration policies. Depending on the write sizes that the filesystem interfaces being used allow clients to specify, a given writeoperation from a client may result in the modification of only a part ofa page rather than the whole page in some cases. If, in a givenimplementation, a physical page is the minimum level of atomicity withrespect to writes supported by the storage subsystem, but write requestscan be directed to arbitrary amounts of data (i.e., writes do not haveto be page-aligned and do not have to modify all the contents of anintegral number of pages), some writes may be treated internally withinthe storage service as read-modify-write sequences. Details regarding anoptimistic conditional-write technique that may be employed for writesthat do not cross page boundaries in some such embodiments are providedbelow. In general, each storage device and/or storage service node maysupport operations for, and/or store data for, a plurality of differentcustomers in at least some embodiments.

In general, metadata and/or data that may have to be read or modifiedfor a single file store operation request received from a customer maybe distributed among a plurality of storage service nodes. For example,delete operations, rename operations and the like may require updates tomultiple elements of metadata structures located on several differentstorage devices. In accordance with the sequential consistency model, inat least one embodiment an atomic metadata operation comprising a groupof file system metadata modifications may be performed to respond to asingle client request, including a first metadata modification at onemetadata subsystem node and a second metadata modification at adifferent metadata subsystem node. Various distributed update protocolsthat support sequential consistency may be used in differentembodiments—e.g., a distributed transaction mechanism described below infurther detail may be used in at least some embodiments for suchmulti-page, multi-node or multi-extent updates. Of course, depending onthe replication strategy being used, each one of the metadatamodifications may in turn involve updates to a plurality of extentreplicas in some embodiments.

In some embodiments, optimization techniques associated with variousaspects of the file storage service, such as the use of object renamingprotocols, load balancing techniques that take connection longevity intoaccount, name space management techniques, client session metadatacaching, offset-based congestion control policies, and the like, may beemployed. Details on these features of the storage service are providedbelow in conjunction with the description of various figures.

File Storage Service Overview

FIG. 1 provides a high-level overview of a distributed file storageservice, according to at least some embodiments. As shown, system 100comprising storage service 102 may be logically divided into at leasttree subsystems: a storage subsystem 130, a metadata subsystem 120 andan access subsystem 110. Each subsystem may comprise a plurality ofnodes, such as storage nodes (SNs) 132A and 132B of storage subsystem130, metadata nodes (MNs) 122A and 122B of metadata subsystem 120, andaccess nodes (ANs) 112A and 112B of the access subsystem 110. Each nodemay, for example, be implemented as a set of processes or threadsexecuting at a respective physical or virtualized server in someembodiments. The number of nodes in any given subsystem may be modifiedindependently of the number of nodes in the other subsystems in at leastsome embodiments, thus allowing deployment of additional resources asneeded at any of the subsystems (as well as similarly independentreduction of resources at any of the subsystems). The terms “accessserver”, “metadata server” and “storage server” may be used herein asequivalents of the terms “access node”, “metadata node” and “storagenode” respectively.

In the depicted embodiment, the storage nodes 132 may be responsible forstoring extents 134 (such as extents 134A and 134 at storage node 132A,and extents 134K and 134L at storage node 132B), e.g., using somecombination of SSDs and rotating disks. An extent, which may for examplecomprise some number of gigabytes of (typically but not always)contiguous storage space at some set of physical storage devices, mayrepresent a unit of storage replication in some embodiments—thus, anumber of physical replicas of any given logical extent may be stored.Each extent replica may be organized as a number of physical pages insome embodiments, with the pages representing the smallest units inwhich reads or writes are implemented within the storage subsystem. Asdiscussed below with respect to FIG. 4, a given file store object (e.g.,a file or a metadata structure) may be organized as a set of logicalblocks, and each logical block may be mapped to a set of pages within adata extent. Metadata for the file store object may itself comprise aset of logical blocks (potentially of different sizes than thecorresponding logical blocks for data), and may be stored in pages of adifferent extent 134. Replicated state machines may be used to manageupdates to extent replicas in at least some embodiments.

The access subsystem 110 may present one or more file system interfacesto clients 180, such as file system APIs (application programminginterfaces) 140 in the depicted embodiment. In at least someembodiments, as described below in further detail, a set of loadbalancers (e.g., software or hardware devices that may be configuredindependently of the storage service itself) may serve as intermediariesbetween the clients of the storage service and the access subsystem. Insome cases, at least some aspects of load balancing functionality may beimplemented within the access subsystem itself. In at least someembodiments the access subsystem nodes 112 may represent serviceendpoints established within the appropriate network fabric that isconcurrently being used by clients 180. As described below with respectto FIG. 3, special network addresses associated with isolated virtualnetworks may be assigned to ANs 112 in some embodiments. ANs 112 mayauthenticate an incoming client connection, e.g., based on the client'snetwork identity as well as user identity; in some cases the ANs mayinteract with identity/authentication services similar to ActiveDirectory Service or Kerberos. Some file system protocols that may besupported by the distributed file storage service 102 (such as NFSv4 andSMB2.1) may require a file server to maintain state, for examplepertaining to locks and opened file identifiers. In some embodiments,durable server state, including locks and open file states, may behandled by the metadata subsystem 120 rather than the access subsystem,and as a result the access subsystem may be considered a largelystateless server fleet that can be scaled up and down as needed. In someembodiments, as described below with respect to FIG. 6, ANs 112 maycache metadata state pertaining to various file store objects, and mayuse the cached metadata to submit at least some internal I/O requestsdirectly to storage nodes without requiring interactions with metadatanodes.

The metadata subsystem 120 may be responsible for managing various typesof file store metadata structures in the depicted embodiment, includingfor example the logical equivalents of inodes, file/directory attributessuch as access control lists (ACLs), link counts, modification times,real file size, logical block maps that point to storage subsystempages, and the like. In addition, the metadata subsystem may keep trackof the open/closed state of the file store objects and of locks onvarious file store objects in some embodiments. The metadata subsystem120 may sequence and coordinate operations so as to maintain desiredfile store object consistency semantics, such as the close-to-opensemantics expected by NFS clients. The metadata subsystem may alsoensure sequential consistency across operations that may involvemultiple metadata elements, such as renames, deletes, truncates andappends, e.g., using the distributed transaction techniques describedbelow. Although the metadata subsystem 120 is logically independent ofthe storage subsystem 130, in at least some embodiments, persistentmetadata structures may be stored at the storage subsystem. In suchembodiments, even though the metadata structures may be physicallystored at the storage subsystem, the metadata subsystem nodes may beresponsible for such tasks as identifying the particular storage nodesto be used, coordinating or sequencing storage operations directed tothe metadata, and so on. In at least some embodiments, the metadatasubsystem may reuse some of the state management techniques employed bythe storage subsystem in some embodiments, such as the storagesubsystem's consensus-based state replication machinery.

Provider Network Implementations of File Storage Service

As mentioned earlier, in some embodiments the distributed storageservice may be implemented using resources of a provider network, andmay be used for file-related operations by applications or clientsrunning at compute instances of the provider network. In someembodiments a provider network may be organized into a plurality ofgeographical regions, and each region may include one or moreavailability containers, which may also be termed “availability zones”herein. An availability container in turn may comprise one or moredistinct locations or data centers, engineered in such a way (e.g., withindependent infrastructure components such as power-related equipment,cooling equipment, and physical security components) that the resourcesin a given availability container are insulated from failures in otheravailability containers. A failure in one availability container may notbe expected to result in a failure in any other availability container;thus, the availability profile of a resource is intended to beindependent of the availability profile of resources in a differentavailability container. Various types of applications may be protectedfrom failures at a single location by launching multiple applicationinstances in respective availability containers. Nodes of the varioussubsystems of the storage service may also be distributed across severaldifferent availability containers in some embodiments, e.g., inaccordance with the availability/uptime goals of the service and/or thedata redundancy requirements for various file stores. At the same time,in some implementations, inexpensive and low latency networkconnectivity may be provided between resources (such as the hosts orstorage devices being used for the distributed file storage service)that reside within the same geographical region, and networktransmissions between resources of the same availability container maybe even faster. Some clients may wish to specify the locations at whichat least some of the resources being used for their file stores arereserved and/or instantiated, e.g., at either the region level, theavailability container level, or a data center level, to maintain adesired degree of control of exactly where various components of theirapplications are run. Other clients may be less interested in the exactlocation where their resources are reserved or instantiated, as long asthe resources meet the client requirements, e.g., for performance, highavailability, and so on.

In at least some embodiments, the resources within a given data centermay be further partitioned into sub-groups based on differences inexpected availability or failure resilience levels. For example, one ormore server racks at a data center may be designated as a lower-levelavailability container, as the probability of correlated failures withina rack may at least in some cases be higher than the probability ofcorrelated failures across different racks. At least in someembodiments, when deciding where to instantiate various components ornodes of the storage service, any combination of the various levels ofavailability containment described (e.g., the region level, the datacenter level, or at the rack level) may be taken into account togetherwith performance goals and durability goals. Thus, for some types ofstorage service components, redundancy/replication at the rack level maybe considered adequate, so in general different racks may be used fordifferent components providing the same function (or storing replicas ofthe same data/metadata). For other components, redundancy/replicationmay also or instead be implemented at the data center level or at theregion level.

FIG. 2 illustrates the use of resources at a plurality of availabilitycontainers 212 of a provider network 202 to implement a file storageservice, according to at least some embodiments. In the embodimentdepicted, three availability containers 212A, 212B and 212C are shown,each of which comprise some number of storage nodes, metadata nodes andaccess nodes of the storage service. Since each availability containeris typically set up so as to prevent correlated failure events thatcross availability container boundaries, the set of storage servicenodes that are assigned to a given file store may typically be spreadacross different availability containers. It is noted that some filestores may have lower availability or durability requirements thanothers, and may therefore be implemented within a single availabilitycontainer in at least some embodiments. In one embodiment, when the filestorage service is set up, a pool of nodes may be established for eachof the three subsystems in each of several availability containers 212,from which specific nodes may be assigned to a given file store asneeded. In other embodiments, instead of establishing pre-configuredstorage service node pools, new nodes may be instantiated as needed.

The collection of ANs, MNs and SNs that collectively implement filestorage for a given file store or file system may be referred to as a“node set” 250 for that file store. In the embodiment shown in FIG. 2,the storage service nodes are multi-tenant, in that a given node of anyof the subsystems may be responsible for handling requests from severaldifferent clients and/or several different customers. It is noted thatin various embodiments, a given customer (e.g., a business entity orindividual on whose behalf a billing account has been established at thestorage service) may set up several different file stores in thedepicted embodiment, and that many different client devices (computingdevices from which programmatic interfaces may be invoked) may be usedto issue file service requests to a single file store by, or on behalfof, a given customer. In at least some embodiments, multiple useraccounts (e.g., one or more user accounts for each of several employeesof a customer business organization) may be set up under the aegis of asingle billing account, and each of the user accounts may submit filestorage requests from a variety of client devices.

Node set 250A of FIG. 2, used for file store FS1 of customer C1,comprises SNs 132A, 132B and 132K, MNs 122A, 122B and 122F, and ANs112A, 112B and 112H, distributed among two availability containers 212Aand 212B. Node set 250B, used for file store FS2 of a different customerC2, comprises nodes in three availability containers 212A, 212B and212C: SNs 132B, 132K, 132L and 132P, MNs 122B 122F, 122G and 122R, andANs 112B and 112M. Node set 250C, used for file store FS3 of customerC1, uses nodes of availability container 212C alone: SNs 132P and 132Q,MNs 122R and 122S, and ANs 112M and 112N. The specific nodes that are tobe used for a given file store may be selected on demand based onvarious factors, e.g., by a placement component of the storage service,and the node set may change over time in view of changing storage spaceneeds, performance needs, failures and the like. A given storage deviceat a single storage node may store data and/or metadata belonging todifferent clients in at least some embodiments. In at least someembodiments, a single extent may comprise data and/or metadata of aplurality of clients or customers.

At least with respect to the SNs, redundancy or replication may beimplemented along several different dimensions for a given file store insome embodiments. As the amount of data in a given file grows, forexample, the various logical blocks of the file may in general be mappedto different logical extents. Thus, file striping may be implemented atthe logical-block level, which may help to improve performance forcertain patterns of I/O requests and may also reduce the time taken torecover a large file in case one of the storage nodes or devices beingused for the file fails. Metadata for the file may also be stripedacross multiple metadata logical extents and managed by multiple MNs insome implementations. Each logical extent (whether for data or metadata)in turn may be replicated across multiple SNs at different availabilitycontainers 212, e.g., using erasure coding or full replication, toachieve the desired degree of data durability. As noted earlier, in atleast one embodiment replication may be implemented across lower-levelavailability containers, e.g., by choosing different racks within thesame data center for different replicas. ANs and MNs may also beorganized into redundancy groups in some embodiments, so that if some ANor MN fails, its workload may be quickly taken up by a different memberof its redundancy group.

In some embodiments, a provider network 202 may support establishment of“isolated virtual networks” (IVNs) on behalf of various customers. AnIVN (which may also be referred to in some environments as a virtualprivate cloud or VPC) set up for a given customer may comprise acollection of computing and/or other resources in a logically isolatedsection of the provider network, over which the customer is grantedsubstantial control with respect to networking configuration. In someembodiments, for example, a customer may select the IP (InternetProtocol) address ranges to be used for the IVN resources, manage thecreation of subnets within the IVN, and the configuration of routetables, gateways, etc. for the IVN. For at least some of the deviceswithin an IVN in some embodiments, the network addresses may not bevisible outside the IVN, at least by default. In order to enableconnectivity between an IVN and the customer's external network (e.g.,devices at the customer's data center or office premises), a virtualinterface that is configured for use with private addresses (and maytherefore be termed a private virtual interface) and a virtual privategateway may be set up. In some embodiments one or more VPNs (virtualprivate networks) may be configured between the customer's IVN andexternal networks (such as the customer's office network or thecustomer's data centers). In at least some embodiments, such VPNs mayutilize secure networking protocols such as IPSec (Internet ProtocolSecurity), SSL/TLS (Secure Sockets Layer/Transport Layer Security), DTLS(Datagram Transport Layer Security) and the like.

In some embodiments, for security or other reasons, access to a givenfile store managed by a distributed storage service may be limited to aspecific set of client devices within one or more IVNs. FIG. 3illustrates a configuration in which network addresses associated withisolated virtual networks 302 are assigned to access subsystem nodes ofa storage service, according to at least some embodiments. As aconsequence of such address assignments, only those clients whosenetwork addresses also lie within the IVN may be able to access the filestore via the ANs 112. As shown, the provider network 202 in FIG. 3includes SNs 132A-132F, MNs 122A-122F, and ANs 112A-112F. Two IVNs 302Aand 302B have been set up in the provider network 202, for customers Aand B respectively. Each IVN includes a number of compute instances(CIs) of virtual computing service 302, at which applications thatrequire file storage services may be run. In addition to the CIs shownwithin the IVNs 302A (e.g., CIs 380A and 380B) and 302B (CIs 380K and380L), other CIs (e.g., 380P and 380Q) may also run on instance hostsoutside the IVNs in the depicted embodiment—thus, not all clients of thefile storage service need necessarily belong to an IVN 302.

In order to enable access to the file storage service from CIs withinIVN 302A, ANs 112A and 112D have been assigned private IP (InternetProtocol) addresses 350A associated with IVN 302A. As a result, clientCIs 380A and 380B of IVN 302A may invoke the file storage serviceinterfaces using addresses 350A, and may be able to rely on variousnetwork isolation and security features already implemented for IVNswhen interacting with the file storage service. Similarly, ANs 112D and112E may be assigned private network addresses of IVM 302B, enablingsecure access from client CIs 380K and 380L of IVN 302B. It is notedthat a given AN (such as 112D) may be assigned more than one networkaddress in at least some embodiments, allowing a single AN's resourcesto be shared by multiple IVNs. In other embodiments, each AN may berestricted to network addresses of no more than one IVN. In addition tothe private addresses, in some embodiments, public network addresses(e.g., IP addresses accessible from the public Internet) may also beused for at least some ANs such as AN 112C, enabling access from CIssuch as 380P or 380Q that are not part of an IVN. In one embodiment,clients located outside the provider network 202 may also be able toaccess the storage service using public IP addresses. In someembodiments, a single (private or public) network address may beassigned to a plurality of ANs 112, so that, for example, incoming workrequests may be balanced across multiple ANs, and AN failover may beimplemented without impacting clients (e.g., clients may continue tosend file store requests to the same address even after a particular ANfails, because the remaining ANs with the same network address maycontinue to respond to client requests).

Logical Blocks, Pages, and Extents

FIG. 4 illustrates a mapping between file storage service objects,logical blocks, and physical pages at one or more extents, according toat least some embodiments. Three logical blocks LB 402A, 402B and 402Chave been configured for a file F1. Logical blocks may also be referredto herein as stripes, as the contents of different logical blocks of agiven object such as file or metadata structure may typically be storedat distinct storage locations. In some embodiments, physical separationof stripes such as stripes A, B and C of file F1 may be enforced—e.g.,no two stripes of a given object may be stored at the same physicalstorage device. In other embodiments, physical separation of stripes mayoccur with a high probability without explicit enforcement, e.g., due tothe use of random or near-random distribution of stripes across largenumbers of physical devices. In at least some embodiments, logical blocksizes may vary within a given file or metadata structure. In otherembodiments, all the logical blocks of at least some storage serviceobjects may be of the same size. The contents of each logical block 402may be stored in one or more physical pages (PPs) 412 of a given dataextent 434 in the depicted embodiment. Thus, for example, contents of LB402 have been written to PPs 412J, 412K and 412L at data extent 434C ofstorage node 132D. Contents of LB 403 are stored in PP 412B within dataextent 434A of storage node 132B, and contents of LB 404 are stored inPP 412F of storage extent 434B at storage node 132C. To simplify thediscussion of the mapping between blocks and pages, extent replicas arenot shown in FIG. 4. At least in the depicted embodiment, the techniquesused for replication of extents may be independent of the techniquesused for mapping blocks to pages.

In at least some embodiments, as described below in further detail,dynamic on-demand allocation may be used for physical storage, inaccordance with which only the set of pages actually needed to store thewrite payload of a given write request may actually be allocated whenthe write request is received. Consider an example scenario in which thelogical block size of a particular LB is 8 megabytes, a fixed page sizeof 64 kilobytes is being used for the extent to which the LB is mapped,and the first write directed to the LB comprises a write payload of 56kilobytes. In such a scenario, only one page (64 kilobytes) of storagespace may be allocated in response to the request in embodiments inwhich on-demand allocation is being used. In other embodiments, physicalstorage for the entire LB may be set aside in response to the firstwrite request directed to the LB, regardless of the write payload size.

When a client writes to a particular file for the first time, a selectedmetadata subsystem node may generate metadata 475 for one or morelogical blocks 402 (e.g., depending on the size of the write payloadrelative to the logical block size, more than one logical block may berequired in some cases). This metadata 475 itself may be stored in oneor more physical pages such as PP 412Q of a metadata extent 464 in thedepicted embodiment. The block sizes and/or page sizes being used formetadata structures may differ from those being used for thecorresponding data in at least some embodiments. In at least oneembodiment, the metadata extents may be stored using a different classor type of storage device (e.g., SSDs) than are used for data (e.g.,rotating disks). In some implementations, at least a portion of themetadata and at least a portion of metadata for the same file storeobject may be stored on the same extent.

In some embodiments, as discussed above, the contents of data extents434 and/or metadata extents 464 may be replicated, e.g., in order tomeet respective data durability requirements. In such embodiments, asdescribed in further detail below, a particular replica of a logicalextent may be chosen as the master replica, and updates to the extentmay be initiated and/or coordinated by the master replica (or thestorage node where the master replica resides), e.g., by propagating theupdates to the required number of replicas from the master beforeindicating that the corresponding update request has succeeded.

The order in which content of a given logical block is written at thestorage device at which any given replica of the extent is stored mayvary—i.e., if two 32-kilobyte physical pages P1 and P2 corresponding toa particular 1-megabyte logical block are located in the order “P1followed by P2” on the disk or SSD, this may not necessarily imply thatthe data in P1 has a lower starting offset within the logical block thanthe data in P2. In some embodiments, pages may be moved (i.e.,rearranged within their storage device) after they are first written,e.g., to facilitate improved sequential read or write performance.Within a given extent or extent replica, physical pages associated withseveral different files may be stored—for example, in metadata extent634, block-to-page maps (or other metadata) of one or more files otherthan F1 may be stored in PPs 412P, 412R and 412S. Similarly, pages 412A,412C, 412D, 412E, 412G, 412H, and 412M may all store contents of filesother than F1. In some embodiments, a large enough number of extents maybe established that the probability of any two logical blocks of thesame file being mapped to the same extent (e.g., to the same replicagroup of extents) may be quite low. In such a scenario, it may bepossible to respond in parallel to concurrent I/O requests directed todifferent logical blocks of the same file, as the requests may bedirected (in most cases) to different storage nodes and differentstorage devices. In at least one embodiment, the storage system may ingeneral tend to distribute logical blocks in an apparently random ornear-random manner among available extents, e.g., by selecting theextent to be used for a particular block based on factors such as theamount of available free space at the time that the particular block isfirst written.

FIG. 5 illustrates a configuration of replica groups 510 for data andmetadata extents, according to at least some embodiments. Two replicagroups 510A and 510B for data extents D1 and D2 are shown, and tworeplica groups 510C and 510D for metadata extents M1 and M2 are shown.Each replica group illustrated comprises two or more replicas atrespective storage devices 532 at respective storage nodes 132 of thestorage subsystem, although in general it may sometimes be the case thattwo physical replicas of the same logical extent are stored on the samestorage device or on different storage devices at the same storage node.

Each replica group 510 is shown as comprising one master replica and oneor more non-master replicas. The master replica may be responsible forcoordinating writes to the members of the replica group, e.g., using areplicated state machine and/or a consensus-based update protocol. Insome embodiments, a replicated state machine and/or a consensus-basedprotocol may also be used for reads as well. The total number ofreplicas in a replication group may vary as a function of the durabilityrequirements for the file data and/or metadata being stored at thereplicas. In FIG. 5, replica 564A is the master replica of group 510A,replica 565B is the master replica of group 510B, replica 575B is themaster replica of replica group 510C, and replica 576B is the masterreplica of replica group 510D. Replica groups 510A and 510C include twonon-master replicas each (replicas 564B and 564C for group 510A, andreplicas 575A and 575C for group 510B). Different types of replicationtechniques may be used in various embodiments, such as erasure-codingtechniques, full replication, or a combination of full and erasure-codedreplicas. In some embodiments, different replication techniques may beused for different file stores.

In at least some embodiments, a variety of different storage devices maybe available for storing extent replicas, such as one or more types ofSSDs and/or individual or arrayed devices based on rotating magneticdisks. In some embodiments, a given storage node 132 may compriseseveral different types of storage devices, while in other embodiments agiven storage node may only have a single type of storage deviceavailable. In the depicted embodiment, storage nodes 132A, 132B and 132Ceach have an SSD device (devices 532B, 532L and 532T respectively at thethree nodes) as well as a rotating disk-based device (532A, 532K and532S respectively). In some implementations, one particular storagedevice technology may be preferred, for storing data extent replicas,metadata extent replicas, or for storing both types of extents as longas space is available. In one implementation, for example, metadataextents may be stored on SSDs when possible, while data extents may bestored on cheaper rotating disks. In some embodiments, data and/ormetadata extents, or portions thereof, may be migrated from one type ofstorage device to another, for example based on usage levels.

Metadata Caching

FIG. 6 illustrates examples of interactions associated with cachingmetadata at access subsystem nodes of a file storage service, accordingto at least some embodiments. As mentioned earlier, in some embodimentsexternal load balancers may be configured to distribute client workloadamong the available access subsystem nodes. In the embodiment depictedin FIG. 6, a service request 644A (such as a write or a read directed toa file) is received from a client 180 at a load balancer 646. The loadbalancer 646 forwards a corresponding service request 644B to a selectedaccess node 112 via a different network connection than was used for theoriginal service request 644A.

The access node 112 may maintain a cache 604 of metadata objectsregarding various file store objects. If metadata sufficient to identifya storage subsystem node 132 that stores the appropriate set of pagescorresponding to forwarded service request 644B happens to be in cache604, the access node may issue read/write requests to the storage node.However, if the metadata is not cached, the access node 112 may submit ametadata request 650 to a selected metadata subsystem node 122, asindicated by arrow 693. As mentioned earlier, the metadata contents mayactually be stored at storage subsystem nodes in some embodiments. Themetadata node 122 (which may comprise, for example, a process executingthe metadata management code) may itself maintain an in-memory set 612of metadata, comprising another cache layer. If the metadata requestedby the access node is not in the in-memory set 612, the metadata nodemay obtain pages 654 containing the metadata from one or more storagenodes 132A, as indicated by arrow 694, and store the metadata in itsin-memory set 612. In some cases, the request 644A from the client mayrequire new metadata to be generated (e.g., if the request was the firstwrite to a file, the metadata node may create metadata entries for thefirst logical block of the file), in which case the metadata node mayensure that the new metadata is safely stored at the storage nodes 132before responding to the request 650 in the depicted embodiment.

At least the portion of the metadata obtained from storage node 132Athat is required for responding to the client's request (termedrequest-relevant metadata 652) may be provided to the access node 112,as indicated by arrow 695. The access node may read the metadata, storeit in cache 604, and submit read or write request(s) 655 to theappropriate storage node(s) 132 identified by the metadata, as indicatedby arrow 696. The storage node(s) 132B may provide a response to theread/write request(s), not shown in FIG. 6, and the access node may insome embodiments respond to the client 180 to indicate whether therequested service operations succeeded or not. The access node 112 maybe able to respond to at least some subsequent client requests using thecached metadata, without having to re-obtain the metadata from themetadata subsystem.

In the depicted embodiment, instead of using explicit cache invalidationmessages, a timeout-based technique may be used for managing potentialstaleness of metadata cache entries at the access node. Thus, the accessnode 112 may use caching timeout setting(s) 608 to determine when toevict any given element of metadata from the cache 604. In someimplementations, a given metadata entry may simply be removed from cache604 after its timeout 608 expires, with no attempt to re-cache it untilit is needed for a different client request. In other implementations,or for some selected types of metadata entries, the access node 112 mayre-request a metadata entry from the metadata node 122 when its cachetimeout expires, or check whether the metadata entry remains valid. Inthe latter scenario, the timeout may be re-set to the original valueeach time that the entry is revalidated or refreshed. At the metadatanode 122, a different type of timeout setting may be used with respectto a given logical block of metadata in the depicted embodiment. Whenthe metadata node 122 initially generates metadata for some file storeobject and stores the metadata in a given logical block of a metadatastructure, a metadata block re-allocation ineligibility timeout periodmay be started, which indicates the minimum amount of time that has topass before that metadata logical block can be re-allocated. (Such ametadata re-allocation may eventually occur, for example, in case theobject whose metadata is stored in the block is deleted.) The blockre-allocation ineligibility timeout setting(s) 614 may typically be setto a longer time period than the cache timeout settings 608 for thecorresponding block metadata. For example, in one implementation, theblock re-allocation timeout value may be two weeks, while the cachetimeout setting may be one day. In such a scenario, the access node 112may re-check the validity of a given block of metadata once every day,while the metadata node 122 may ensure that that block is not re-usedfor some other purpose before two weeks have passed since the initialallocation of the block.

In some embodiments, instead of using a timeout-based mechanism, anexplicit lease or lock may be used for metadata entries cached at theaccess node. In at least one embodiment, an explicit cache invalidationmechanism may be used, in which for example the metadata node 122 maysend out invalidation messages when some element of metadata is nolonger valid. In one embodiment, the metadata subsystem may mark a blockof metadata “invalid” or “inaccessible” in response to metadata changes.When an access node attempts to use invalid cached metadata to accessdata blocks, an error message indicating that the metadata is invalidmay be returned by the metadata subsystem or the storage subsystem tothe access node. Thus, the cached metadata may be invalidated implicitlyas a result of such error messages. Various combinations oftimeout-based, lock/lease-based, implicit and explicitinvalidation-based strategies may be used in different embodiments formetadata cached at the access nodes.

In some of the interactions depicted in FIG. 6, such as those indicatedby the arrow labeled 693, 694 and 696, some components of the storageservice may act as clients of other components. For example, the accessnode 112 may send internal requests (i.e., requests that are generatedwithin the storage service and use network paths that are not directlyaccessible to customers of the storage service) to the metadata node(arrow 693), acting as a client of the metadata node. Similarly, boththe metadata node and the access node may send internal requests tostorage nodes 132, acting as clients of the storage nodes. In someembodiments, the various subsystems may implement internal APIs that canbe invoked by other components of the storage service to enable suchinteractions. A storage node 132 may, for example, respond in the sameway whether a particular storage service API was invoked from an accessnode 112 or from a metadata node 122. Thus, at least in someembodiments, storage service nodes may be agnostic with respect to thesources from which they are willing to receive internal requests.

File Store Policies

In some embodiments, clients may be granted substantial flexibility tocontrol various aspects of the behavior of the file storage service withrespect to specific file stores. For example, one or more administrativeAPIs may be implemented to allow clients to set or modify thedurability, performance, availability or other requirements for aparticular file store, which may differ from the correspondingrequirements for other file stores created on behalf of the same clientor other clients. FIG. 7 illustrates examples of the use of distinctcombinations of policies pertaining to data durability, performance, andlogical-to-physical data mappings for file stores, according to at leastsome embodiments.

As shown in columns 704 and 714, the durability policies for data andmetadata respectively for a given file store such as FS1 may differ, andthe durability policies used at different file stores such as FS1 andFS2 may differ for either data, metadata or both. For FS1, 10-way fullreplication is used for metadata (10 full copies of each page ofmetadata are maintained), while 12/6 erasure coding is used for datadurability (12 erasure coded copies are stored of each data page, ofwhich 6 are needed to reconstruct the contents of the page). Performancegoals/requirements for the metadata and data of file stores FS1 and FS2are indicated in columns 706 and 716 respectively. The performance goalsmay be expressed in various units, e.g., units for latency or responsetime (indicated by the label “resp time” in columns 706 and 716) versusunits for throughput (indicated by the label “tput”), and in some casesdifferent sets of requirements may be specified for reads (indicated bythe label R in columns 706 and 716) than for writes (indicated by thelabel W). The performance goals may be used, for example, to select thetypes of storage devices that should be used for a given file store'smetadata or data.

Different approaches may be used for allocating storage space forstorage objects for respective file stores in the depicted embodiment.For example, as indicated in column 708, a fixed logical block size of512 kilobytes and a policy of dynamic page allocation is used for FS1metadata, while for FS2 metadata, physical storage for one-megabytelogical blocks may be allocated statically. As shown in column 718, forFS1 data, a varying logical block size may be used, with the first fewlogical blocks of a given file being set to 1 kilobyte, 1 kilobyte, 2kilobytes, 2 kilobytes, etc., with the logical block size graduallyincreasing as the file grows. For FS2 data, in contrast, fixed-size4-megabyte logical blocks may be used. The physical page sizes used formetadata may be set as follows (column 710): 8 kilobytes for FS1 and 16kilobytes for FS2. For data, as shown in column 720, the page size maybe set equal to the logical block size for FS1, while the page size maybe set to 32 kilobytes for FS2. Respective metadata cache-relatedsettings for FS1 and FS2 are shown in column 712, including metadatacache timeouts and the block reallocation ineligibility timeoutsdiscussed above with reference to FIG. 6. In some embodiments, e.g., inorder to decrease implementation complexity of the file storage service,only a discrete set of options may be supported for durability policies,block and page sizing policies, and the like. Other types of policies,such as availability-related or uptime requirements, file store spacelimits, and the like, may also be set differently for different filestores in some embodiments. In at least one embodiment, clients may beable to choose from among a plurality of pricing policies on aper-file-store basis as well—e.g., some clients may select astorage-space-usage-based pricing policy, while other clients may selecta file system API-count-based pricing policy.

Methods of Implementing a Scalable File Storage Service

FIG. 8a is a flow diagram illustrating aspects of configuration andadministration-related operations that may be performed to implement ascalable distributed file system storage service, according to at leastsome embodiments. As shown in element 801, an initial set of M emptyextents may be established for data and/or metadata, e.g., at Ndifferent storage subsystem nodes of a distributed file storage serviceduring a service initialization procedure. The storage service may beset up to implement file storage operations on behalf of clientapplications running on compute instances of a virtual computing serviceestablished at a provider network in some embodiments. In variousembodiments, each storage node may comprise a plurality of extents,e.g., M may be larger than N. In embodiments in which extent contentsare replicated for data durability, each of the M empty extents may becapable of storing a respective replica of the contents of a logicalextent. Each storage node may comprise one or more storage devices,including for example some number of rotating disk-based devices and/orsolid-state store devices. A given extent may be incorporated within asingle storage device in some embodiments, or may be spread overmultiple storage devices in other embodiments. In one embodiment, allthe extents may be of the same size, e.g., based on a configurableparameter associated with the storage service. In other embodiments,different extents may have different sizes, and/or the size of an extentmay change over time. The total number of extents in a giveninstantiation of the storage service may vary over time—e.g., as thesize of the metadata and data grows, more storage devices and/or moreextents may be deployed. The extents may represent a unit of recoverywith respect to data and metadata of the storage service in someembodiments—e.g., each extent may be replicated based on durabilitypolicies or settings, using erasure coding, full replication, or somecombination of replication techniques. Each extent replica group (i.e.,a group of replicas of the same logical data or metadata extent) mayinclude at least one replica designated as a master replica whosestorage node (which may also be referred to as a master node or a leadernode with respect to the logical extent) is responsible for coordinatingupdates among the group members. In some embodiments, decisionsregarding master selection and/or membership of replica groups may bedeferred until the first object of a file store is written. In at leastsome implementations, the extents may be multi-tenant—e.g., each extentmay store data or metadata of a number of different clients orcustomers.

Some number of access subsystem nodes may be established initially toenable access to at least a particular file store FS1 (element 804) inthe depicted embodiment. For example, in an embodiment in which the filestore clients comprise compute instances of an isolated virtual network(IVN), private IP addresses accessible only from within the IVN may beassigned to the P access subsystem nodes. Public IP addresses may alsoor instead be assigned to some or all of the access subsystem nodes insome embodiments. In some embodiments, a pool of partiallypre-configured access subsystem nodes may be set up, and specific accessnodes may be assigned for particular file stores from the pool; in otherembodiments, access nodes may be instantiated on demand. A given networkaddress may be assigned to more than one access subsystem node in atleast one embodiment.

In some embodiments, a set of Q metadata nodes may be assigned to thefile store FS1 upon file store creation. In other embodiments, metadatanodes (which may also be selected from a pre-configured pool, or may beinstantiated dynamically) may be assigned to FS1 on-demand, e.g., whenthe first write request to an object of FS1 such as a file or adirectory is received (as described below with respect to FIG. 8b ).Administrative components of the file storage service may monitor theperformance and/or health status of various nodes of the accesssubsystem, the metadata subsystem, and the storage subsystem in thedepicted embodiment (element 807). Records of the completed orsuccessful file store operations performed on behalf of any given clientmay be stored, and such records may be later used to generateusage-based billing amounts for the client in the depicted embodiment.In response to an analysis of observed performance metrics and/or healthstatus changes, nodes may be dynamically added or removed from any ofthe subsystems without affecting the population of the other layers, andwithout impacting the stream of incoming file storage requests (element810). E.g., in response to a detection of a possible performancebottleneck at the access subsystem, or a detection of a failed orunresponsive access subsystem node, more access subsystem nodes may beinstantiated without affecting either of the other subsystem nodes. Insome cases, if the resource utilization (e.g., CPU or storageutilization) at one or more nodes remains below a threshold for someperiod of time, such nodes may be eliminated and their workload may bedistributed among other nodes. Thus, each of the subsystems may beindependently scaled up or down as needed.

FIG. 8b is a flow diagram illustrating aspects of operations that may beperformed in response to client requests at a scalable distributed filesystem storage service, according to at least some embodiments. Inresponse to a create (e.g., an invocation of an “open” API) or a firstwrite request directed to a file of file store FS1, for example, spacemay be allocated at one or more selected metadata extents and dataextents (element 851). In the depicted embodiment, the metadatasubsystem may store the metadata contents at storage subsystem nodes,e.g., the storage capabilities of the storage subsystem may be re-usedfor metadata instead of implementing a separate storage layer strictlyfor metadata. In other embodiments, a separate storage subsystem may beused for metadata than is used for data. In embodiments in whichreplication is being used to achieve desired data durability, space maybe allocated at a plurality of metadata and/or data extents, e.g., forall the members of the appropriate extent replica groups. A particularextent may be selected to allocate one or more pages to respond to thefirst write based on various factors in different embodiments—e.g.,based on how full the extent currently is, based on the performancecharacteristics of the extent relative to the performance requirementsof the object being created, and so on. In at least some embodiments,the current “spread” of the objects of the file store may also be takeninto account when selecting an extent—e.g., the storage subsystem mayattempt to reduce the probability of “hot spots” by avoiding storing toomany blocks of a given file store's data or metadata at the same extentor at the same storage node.

As additional writes are directed to objects within FS1, additionalspace may be allocated for data and/or metadata, e.g., at other storagesubsystem nodes based on applicable striping policies (i.e.,logical-block-to-physical-page mapping policies), and additionalmetadata nodes may be configured as needed (element 854). The nodes ofeach of the three subsystems—the storage subsystem, the access subsystemand the metadata subsystem—may be configured to support multi-tenancy inat least some embodiments—e.g., each storage service node may handlestorage requests from, or store data/metadata of, several differentclients at the same time. The clients may not be aware that the sameresources that are being used for their storage requests are also beingused for requests from other clients. Each storage service node maycomprise, for example, one or more processes or threads that may beexecuted using hosts or servers of a provider network in someembodiments.

Over time, the metadata corresponding to a given file store object suchas a directory or a file may end up being distributed across severaldifferent extents at several different storage nodes. Some file storageoperations (e.g., rename operations or delete operations) may requiremodifications to metadata at more than one extent, or at more than onestorage node. In response to a request for such an operation, thestorage service may perform an atomic update operation that includeschanges at more than one metadata page or more than one metadata extent(element 857) in a manner that supports or enforces sequentialconsistency. Any of a number of different types of consistencyenforcement techniques may be used in different embodiments, such as adistributed transaction technique or a consistent object renamingtechnique, which are both described in further detail below.

FIG. 9 is a flow diagram illustrating aspects of operations that may beperformed to implement a replication-based durability policy at adistributed file system storage service, according to at least someembodiments. As shown in element 901, values for each of a set ofdurability-related parameters that are to be used for the data and/ormetadata of a given file store object F1 may be determined, e.g., at thetime that the object is created. The parameters may include replicacounts—e.g., the number of replicas of each page, and therefore eachextent, that stores contents of the object or contents of metadatarelated to the object in some embodiments. The replication strategy(e.g., whether full replication is to be used, erasure-coded replicationis to be used, or some combination of such techniques is to be used),and/or the placement of the replicas among the available data centerresources may also be specified as parameters in some embodiments. Forexample, in some embodiments in which the storage service includes aplurality of availability containers, at least one replica may be placedwithin each of K availability containers. An appropriate set of extentreplicas may then be identified in accordance with the parameters(element 904). In some embodiments, the specific physical extents may bechosen based on an analysis of the amount of free space available atvarious candidates, recent workload levels at the extents or theircontaining storage servers, locality with respect to expected sources ofclient requests, the “spread” of the file store for which space is beingallocated as described earlier, or based on other metrics. One of thereplicas may be designated as the master replica, and its storage nodemay be designated as a leader responsible for coordinating variousoperations such as writes directed to the file store object among themembers of the replica group (element 907). In at least someembodiments, the particular storage node chosen as a leader forcoordinating data writes to a given file store object may also beselected as the leader for coordinating metadata writes for that filestore object (even though at least some of the metadata may be stored atdifferent storage nodes than the data).

In response to a particular write request directed to a logical block ofthe file store object from a client, an internal write request may bedirected to the master extent replica of the logical extent to whichthat logical block is mapped (element 910). Thus, for example, theaccess node that received the client's request may first have toidentify the master extent replica for the logical block, e.g., usingmetadata extracted from the appropriate metadata subsystem node, andthen direct an internal write request to the storage node storing themaster replica. In response to receiving the internal write request, theleader node may initiate interactions of a consensus-based statemanagement protocol to replicate the write payload among the replicagroup members (element 913). In at least some implementations, theconsensus-based protocol may be used to replicate log records of statechanges, and a representation of the state itself may be replicatedusing erasure cording or using full replicas. If the write is committedas a result of the protocol interactions, e.g., if the write succeeds ata quorum of the replica group members, in some embodiments therequesting client may eventually be informed that the write requestsucceeded. In other embodiments, at least for some types of operationsand some file system protocols, clients may not necessarily be providedan indication as to whether their request succeeded or not. Instead, forexample, the clients may be expected to retry operations that appear notto have succeeded.

FIG. 10 is a flow diagram illustrating aspects of operations that may beperformed to cache metadata at an access subsystem node of a distributedfile system storage service, according to at least some embodiments. Asshown in element 1001, service endpoint addresses that allow clients tosubmit file store-related requests to a set of access subsystem nodes ofa distributed file storage service may be configured. In someembodiments, as discussed earlier, private IP addresses that areaccessible only within an isolated virtual network may be assigned forthe access nodes. In other embodiments, public IP addresses that can beaccessed by non-IVN clients may also or instead be used. The accesssubsystem nodes may be configured to respond to various types ofcommands, system calls, or API invocations conforming to one or moreindustry-standard file system protocols (e.g., one or more versions ofNFS, SMB, CIFS, and the like). In some embodiments a given accesssubsystem node may be capable of responding to commands formatted inaccordance with a plurality of such standards or protocols. In oneembodiment, proprietary file system interfaces may also or instead besupported.

A command (e.g., a create, read, write, modify, reconfigure, or deletecommand) formatted in accordance with one of the APIs/protocols anddirected to a particular file store object F1 may be received at aparticular access node AN1 (element 1004). AN1 may perform a set ofauthentication and/or authorization operations (element 1007), e.g.,based on the network identity (e.g., the source network address), useridentity (e.g., a user account identifier), or other factors to decidewhether to accept or reject the command.

If the command passes the authentication/authorization checks, AN1 mayidentify a metadata node MN1 from which metadata pertaining to F1, to beused to implement the requested operation, is to be obtained (element1010). The access node AN1 may then submit a metadata request to MN1(element 1013). In some embodiments, the identification of theappropriate metadata node may itself involve the submission of anotherrequest, e.g., to a metadata node that manages mappings between storageobjects and other metadata nodes. A block of metadata pertaining to thefile store object F1 may then be obtained at AN1. AN1 may store themetadata in a local metadata cache (element 1016), with a cache timeoutsetting indicating when the block of metadata is to be discarded (aspotentially stale) or has to be re-validated. In at least someembodiments, the cache timeout interval may be set to a value smallerthan a metadata block re-allocation timeout setting used at the metadatanode to determine when it is acceptable to re-use to recycle the blockof metadata for other purposes (e.g., to store metadata for a differentfile store object F2 in the event that F1 is deleted).

AN1 may use the received block of metadata to identify the particularstorage node SN1 to which an internal read/write request is to bedirected, and submit the internal request accordingly (element 1019).Prior to the expiration of the cache timeout, AN1 may re-use the cachedblock of metadata to issue additional internal requests that may resultfrom further invocations of the APIs/protocols (element 1022). At theend of the cache timeout period, the block of metadata may be deleted ormarked as invalid in some embodiments. In at least one embodiment,instead of merely discarding the metadata, the access node mayre-validate it, e.g., by sending another request to the metadata nodefrom which the metadata was obtained.

Conditional Writes for Single-Page Updates

As discussed earlier, in at least some embodiments the file storageservice may be designed to handle large numbers of concurrent operationsfrom hundreds or thousands of clients, potentially directed to thousandsof file store objects. Traditional locking-based mechanisms to ensureatomicity and consistency may not work in such high-throughputhigh-concurrency environments, as the locking system itself may become abottleneck. Accordingly, one or more optimistic schemes may be used forconcurrency control in at least some embodiments, as described below.First, a concurrency control mechanism for single-page writes (i.e.,write requests whose modifications are limited to a single page of asingle logical extent) is described, and later a distributed transactionmechanism that can be used to implement multi-page writes as atomicoperations is described.

In at least some implementations, as also described above, the physicalpages used for storing data and metadata of a given file store maydiffer in size from the logical blocks of the corresponding objects,while write operations may in general be directed to arbitrary offsetsand have write payloads of arbitrary sizes. For example, for at leastsome file system protocols/APIs, from the perspective of an end user ofa file, a single write to the file may modify data starting at anydesired byte-level offset within the file, and may modify (or write forthe first time) any number of bytes starting from that byte-leveloffset. The storage subsystem of the file storage service may, however,treat physical pages as the units of atomicity in some embodiments—e.g.,to reduce implementation complexity, a page may represent the minimumgranularity supported by the storage subsystem's internal read and writeAPIs. Thus, there may a mismatch between the flexibility of the filestore APIs exposed to the end users, and the constraints on the internaloperations supported by the storage subsystem. Accordingly, the clientsof the storage subsystem (e.g., the access nodes or the metadata nodes)may be forced to translate arbitrary write requests into page-levelinternal write operations in such embodiments. In at least someembodiments, at least some internal metadata manipulations that may notresult directly from external client requests may in some cases need tomodify only a small portion of a given page of metadata. Such metadatawrite requests may also have to be implemented at page granularity.

Accordingly, at least some write operations directed to physical pagesmay be implemented as read-modify-write sequences. FIG. 11 illustratesexamples of read-modify-write sequences that may be implemented at afile storage service in which write offsets and write sizes maysometimes not be aligned with the boundaries of atomic units of physicalstorage, according to at least some embodiments. As shown, a file storeobject (such as a file or a metadata structure) may be organized as aset of logical blocks (LBs) 1102, including LB 1102A, 1102B and 1102C.Each logical block may be mapped to a set of pages within an extent(e.g., one logical extent and a plurality of physical extent replicas)of a storage subsystem, where the pages represent the units of atomicitywith respect to the storage subsystem's APIs. For example, logical block1102A is mapped to physical pages (PPs) 1112A, 1112B, 1112C and 1112D ofextent 1164 in the depicted embodiment.

In response to a particular write request 1160, only a portion of asingle page (such as the shaded portion of PP 1112A in the case of writerequest 1160A, and the shaded portion of PP1102D in the case of writerequest 1160B) may actually have to be changed. However, because thestorage subsystem APIs may not permit partial-page writes in thedepicted embodiment, each of the write requests shown may be translatedinto a read-modify-write sequence directed to the corresponding physicalpage. Thus, the client (e.g., an access node or metadata node thatissued the internal write requests 1160) may determine that to implementthe intended partial write, it must first read the targeted page, applythe desired changes, and then submit a write of the entire page. Forwrite request 1160A, the read-modify-write sequence RMW 1177A may beimplemented, comprising a read of page 1112A, a local modification ofthe contents of the page 1112A at the client, and a write of the entirepage 1112A. For write request 1160B, RMW 1177B may be implemented,involving a read of page 1112D, followed by a modification and then awrite of the entire page 1112D.

Given the possibility of concurrent or near-concurrent updates beingrequested to the same physical page, the storage service may have toensure that contents of a given physical page has not been modifiedbetween the read of an RMW sequence and the write of the RMW sequence.In at least some embodiments, a logical timestamp mechanism, which maybe implemented for replicated state management at the storage subsystem,may be used to ensure such sequential consistency as described below.

As mentioned earlier and shown in FIG. 5, replica groups of logicalextents may be used in at least some embodiments to achieve the desiredlevel of data durability. FIG. 12 illustrates the use of consensus-basedreplicated state machines for extent replica groups, according to atleast some embodiments. For logical extent E1, four extent replicas areshown in the depicted embodiment: master replica 1264A at storage node132, and non-master replicas 1264B, 1264C, 1264D at respective storagenodes 132B, 132C and 132D. For a different logical extent E2, masterextent replica 1265C at storage node 132D and two non-master replicas1265A (at storage node 132A) and 1265B (at storage node 132B) are shown.A consensus-based replicated state machine 1232A may be used by node132A (the node at which the master replica is stored) to coordinatevarious operations on the E1 replicas, and a different consensus-basedreplicated state machine 1232B may be used by node 132D (the node atwhich master replica 1265C resides) to coordinate operations on E2replicas.

State machine 1232A may utilize a logical clock 1222A in the depictedembodiment, and state machine 1232B may utilize a different logicalclock 1222B. The logical clock may be used to indicate the relativeordering of various operations managed using the corresponding statemachine, and may not be related directly to a wall-clock time or anyparticular physical clock in at least some embodiments. Thus, forexample, a particular logical clock value LC1 may be associated with thecommit of a write operation coordinated using the state machine 1232A,and a different logical clock value LC2 may indicate when a response toa read operation was provided from the replica group. If LC1<LC2 in thisexample, this would indicate that from the perspective of the storagesubsystem, the write operation was completed prior to the readoperation. The values of the logical clock may also be referred toherein as “logical timestamps” or as “operation sequence numbers” (sincethey may indicate the sequence in which various read or write operationswere completed using the associated replicated state machine). In someimplementations an integer counter implemented at the storage node atwhich the master replica is resident may be used as a logical clock, andthat storage node may be responsible for changes to the clock's value(e.g., the counter may be incremented whenever a read or write operationis completed using the state machine).

The storage nodes may associate logical timestamp values obtained fromthe state machines 1232 with the read and write requests of the RMWsequences described above, and may use the logical timestamps to decidewhether a particular single-page write is to be committed or aborted invarious embodiments. FIG. 13 illustrates example interactions involvedin a conditional write protocol that may be used for some types of writeoperations, according to at least some embodiments. As shown, as part ofa read-modify-write sequence corresponding to a particular writeoperation, a client 1310 of the storage subsystem (such as an accessnode or a metadata node) may submit a read page request 1351 to astorage node 132 (e.g., the node at which the master replica of theextent to which the page belongs is stored). The storage node mayprovide a read response 1352 that comprises the contents of therequested page as well as a read logical timestamp (RLT) assigned to theread operation. The RLT may be obtained, for example, from thereplicated state machine being used for the extent.

Continuing with the RMW sequence, the storage subsystem client 310 maysubsequently submit a write request 1361 for the entire page to thestorage node 132, and may include the RLT that was included in the readresponse. The storage node may determine whether the page has beensuccessfully updated since the RLT was generated. If the page has notbeen updated since the RLT was generated, the requested write may becompleted and a write response 1362 indicating success may be providedto the storage subsystem client. If the page has been updated as aconsequence of another intervening write request since the RLT wasgenerated, the write request may be rejected. Accepting such a writerequest may in some cases lead to data inconsistency, because, forexample, the specific data D1 to be written in response to a given writerequest may be dependent on a value R1 read earlier from the page, andthat value R1 may have been overwritten by the intervening write. Insome implementations, if the write request from client 1310 is rejected,a write response 1362 indicating that the write was aborted may beprovided to the client; in other implementations no write response maybe provided. If the write is aborted, the client 1310 may initiate oneor more additional RMW sequences for the same page in some embodiments,e.g., until the write eventually succeeds or until some threshold numberof write attempts fails.

In order to detect whether an intervening write to the same page hassucceeded since the RLT was generated, in some embodiments write logbuffers that store write logical timestamps may be implemented atstorage nodes 132. FIG. 14 illustrates example write log buffers thatmay be established to implement a conditional write protocol, accordingto at least some embodiments. In the depicted embodiment, a respectivecircular write log buffer 1450 is maintained for each logical extent,e.g., at the storage node where the master replica of the extent isstored. Circular buffer 1450A is maintained for extent E, by the storagenode 1432A managing E1's master replica 1410A, and circular buffer 1450Bis maintained by the storage node 1432B at which E2's master replica1410B is stored. Each circular buffer comprises a plurality of write logrecords 1460, such as records 1460A, 1460B, 1460C and 1460D in buffer1450A and records 1460K, 1460L, 1460M and 1460N in buffer 1450B. Eachlog entry 1460 in the depicted embodiment comprises a respectiveindication of a committed (i.e., successful) page write, indicating thepage identifier that was written to, the logical timestamp associatedwith the completion of the write, and the client on whose behalf thewrite was performed. Thus, in buffer 1450A, records 1460A-1460D indicatethat pages with identifiers 1415A-1415D respectively were written to, inan order indicated by respective write logical timestamps 1417A-1417D onbehalf of clients with respective identifiers 1419A-1419D. Similarly,buffer 1450B indicates that pages with identifiers 1415K-1415Nrespectively were written to, in an order indicated by respective writelogical timestamps 1417K-1417N on behalf of clients with respectiveidentifiers 1419K-1419N. In at least some embodiments, the write logbuffers may be maintained in main memory for fast access. In at leastone implementation, the write logical timestamp of a given record 1460may be implicitly indicated by the relative position of that recordwithin the buffer. Thus, in such an implementation, explicit values ofwrite logical timestamps need not be stored in the buffer. In someembodiments the log buffers may be stored in persistent memory, and mayhave indexes set up for speed retrieval by timestamp value, by pageidentifier, and/or by client identifier. In various embodiments, writelogical timestamp information similar to that shown in FIG. 14 may bemaintained at different granularities—e.g., either at the physical pagegranularity, at the extent granularity, or at some other level.

When the storage node 1432 has to determine whether a particular writeof a read-modify-write sequence is to be accepted or rejected, and thewrite request includes the read logical timestamp (RLT) of the readoperation of the sequence, it may inspect the write log buffer to seewhether any writes with larger logical timestamps than the RLT haveoccurred to the same page. For example, if the RLT value correspondingto a write request of an RMW sequence for a page P1 is V1, the minimumwrite logical timestamp among the records 1460 is V2<V1, and there is norecord in the buffer with a value V3>V1, then the storage node 1432 mayconclude that no intervening write to page P1 has occurred, and thewrite of the RMW may accepted. If there is an entry with a write logicaltimestamp V3>V1 for page P1, the write may be rejected or aborted in thedepicted embodiment. If the minimum write logical timestamp V2 among therecords in the circular buffer 1450 is greater than V1, this mightindicate that some writes directed to P1 may have succeeded since theRLT was generated but may have had their write log records overwritten(e.g., due to buffer space limitations), so at least in some embodimentsthe write request for P1 may also be rejected in such a scenario. If thewrite request of the RMW is accepted, a new write log record 1460 may beadded to the circular write log buffer (potentially overwriting anearlier-generated log record) with a write logical timestampcorresponding to the commit of the write. (It is noted that depending onthe number of replicas that have to be updated, and the replicationprotocol being used, it may take some time before the modification ispropagated to enough replicas to successfully complete or commit thewrite.)

Circular buffers may be used in the depicted embodiment so that thetotal amount of memory used for the buffers remains low, and older writelog records gradually get overwritten by more useful recent write logrecords. As the write operation of a particular read-modify-writesequence is typically expected to be performed fairly quickly after theread, older write log records may typically not be of much help indeciding whether to commit or abort a write of an RMW sequence. However,as discussed above, in some scenarios it may be the case that writes tothe extent are so frequent that potentially useful write log records mayget overwritten within the circular buffer. In some embodiments, thestorage service may keep track of the number of writes that are rejectedbecause of such overwrites, i.e., the write rejection rates causedspecifically as a result of comparisons of read logical timestamps withearliest logical timestamps of the buffer (and subsequent determinationsthat the read logical timestamp is before the earliest logicaltimestamp) may be monitored. In some such embodiments the size of thecircular log buffers may be modified dynamically—e.g., it may beincreased in response to a determination that the write rejection ratesresulting from buffer space constraints has exceeded some threshold, orit may simply be increased during heavy workload periods. Similarly,buffer sizes may be decreased during light workload periods or inresponse to a determination that the rejection rates attributable tobuffer size constraints are lower than some threshold. In someembodiments other types of buffers (i.e., buffers that are not circular)may be used. In at least one embodiment the client identifiers may notbe stored in the write log buffers. In some embodiments buffers similarto those shown in FIG. 14 may be used to record reads as well as writes.In at least one embodiment, the length of the buffer may be dynamicallyadjusted based on the timing of the reads of outstandingread-modify-write sequences. For example, if the read of a particularRMW sequence occurs at time T1, and the buffer becomes full at some timeT2 before the corresponding write request of that sequence is received,the buffer size may be increased (e.g., within some maximum lengththreshold and/or some maximum time threshold) in an attempt to make thecorrect decision regarding accepting the corresponding write. In somesuch scenarios, when the corresponding write is received, say at timeT3, the buffer size may be reduced again to its previous length.

In at least one embodiment, the storage service may maintain versioninginformation at the per-page level, and use the versioning information todecide whether a write of an RMW should be accepted or not. For example,instead of maintaining a log buffer of write operations at theper-extent level, in one such versioning approach, log entries may bemaintained at the per-page level, so that it becomes possible todetermine whether a write of an RMW is directed to the same version asthe corresponding read. If a new version has been created since theread, the write may be rejected.

FIG. 15 is a flow diagram illustrating aspects of operations that may beperformed to implement a conditional write protocol at a distributedfile system storage service, according to at least some embodiments. Asshown in element 1501, a determination may be made at a client C of astorage subsystem (such as an access node or a metadata node) that inorder to implement a particular file store operation, aread-modify-write sequence on a particular page P is to be implemented.In some embodiments, all single-page writes may be translated intoread-modify-write operations by default, even if the entire page isbeing modified; hence, in such embodiments, any write to any page may betranslated into a RMW sequence, and a determination regarding whether anRMW is needed or not may be required. In other embodiments, writes thatmodify the whole page may not require translation to RMW sequences,while writes that modify only part of a page may be translated to RMWsequences.

As shown in element 1504, as part of the RMW sequence, a read requestdirected to P may be received from C at a storage node SN1 (e.g., thenode at which the master replica of the extent to which P belongs isstored). A read logical timestamp RLT corresponding to the read request,indicating the order on which the read is performed relative to otherreads and writes at the same extent, may be obtained (element 1507),e.g., from a replicated state machine being used to manage P's extent.The RLT may be provided to the client C that submitted the read request.

Subsequently, a write request WR1 of the RMW sequence directed to page Pmay be received from C at SN1 (element 1510). The write request mayinclude the RLT value that was provided to C in the read response ofelement 1507, as well as the write payload (i.e., the modification to beapplied to P). The storage node SN1 may determine whether the page P hasbeen modified since the RLT was generated, e.g., by inspecting contentsof a write log buffer that stores the logical timestamps associated withrecent successful writes. If it is determined that P has not beenmodified since RLT was generated (element 1513), the write may beimplemented by making the appropriate modifications to P and propagatingthe modifications to the appropriate number of replicas (element 1516).A write logical timestamp corresponding to the completion of the writemay be stored in a write log buffer in the depicted embodiment, and atleast in some embodiments an indication that the write completed may besent to the client that issued the RMW sequence. In some implementationsthe write logical timestamp may be provided to the client as part of thecompletion indication. If it is determined that P has been modifiedsince RLT was generated (also in operations corresponding to element1513), the write may be rejected and in some embodiments a “writeaborted” response may be sent to the client.

Distributed Transactions Using Ordered Node Chains

The conditional write technique described above may be used for ensuringsequential consistency among single-page write operations in variousembodiments. However, for some types of operations of a distributed filestorage service (such as deletions, renames and the like), multiplepages of metadata and/or data may have to be modified atomically—thatis, either all the changes to all the pages involved have to becommitted, or all the changes have to be rejected. A higher-leveloptimistic consistency enforcement mechanism involving distributedtransactions may be employed for this purpose in at least someembodiments. To implement a distributed transaction in such anembodiment, a coordinator node (e.g., one of the metadata and/or storagenodes involved) may be selected. The coordinator may identify thestorage nodes that are to participate in the changes, determine asequence in which the individual page-level changes are to be examinedfor acceptance or rejection at respective storage nodes, and theninitiate an ordered sequence of operations among the storage nodes inwhich each of the nodes can make a respective commit/abort decision fortheir page-level changes. If all the participants decide that theirlocal changes are committable, the transaction as a whole may becommitted, while if any one of the participants determines that theirlocal page-level changes cannot be committed, the transaction as a wholemay be aborted. Details regarding various aspects of the operations ofthe coordinator and the participant nodes are provided below.

FIG. 16 illustrates an example message flow that may result in a commitof a distributed transaction at a file storage service, according to atleast some embodiments. A determination may be made that a particularfile store operation requires multiple pages to be written, e.g., eitherat an access subsystem node or at a metadata node. A correspondingmulti-page write request 1610 may be generated. The set of pages to bemodified may be termed the “targeted pages” of the transaction herein. Aparticular node of the storage service (which may be either an accessnode, a metadata node, or a storage node in various embodiments) may beselected as a coordinator node 1612 for a distributed transaction toatomically implement the set of writes to the targeted pages. Thecoordinator may identify the set of pages that are to be modified andthe set of storage nodes (which may include itself if the coordinator isa storage node) at which page-level changes are to be initiated orperformed (e.g., the set of storage nodes at which master replicaextents containing the targeted pages are stored). Any of a variety oftechniques may be used to select the coordinator node—e.g., in someembodiments, the storage node at which a randomly-selected page of theset of pages to be modified resides may be selected as the coordinator,while in other embodiments the workload levels at candidate coordinatornodes may be taken into account, and an attempt may be made todistribute the work associated with transaction coordination among thestorage nodes of the service.

In at least some embodiments, a sequence in which the pages targeted formodifications should be locked may be determined by the coordinator 1612in accordance with a deadlock avoidance technique. For example, adeadlock analysis module may be provided the identifiers of the pagesand extents to be modified in the transaction, and the deadlock analysismodule may sort the identifiers based on some selected sort order (e.g.,a lexicographic sort order based on a concatenation of extent ID, pageID and/or other factors) to determine the locking order. The same sortorder may be used consistently across all the distributed transactionsfor the file store, and as a result locks for any given pair of pages P1and P2 may always be requested in the same order. For example, if thedeadlock analysis module indicates that a lock on P1 should be acquiredbefore a lock on P2 for transaction Tx1, it would never indicate that alock on P2 should be acquired before a lock on P1 for any othertransaction Tx2, thus avoiding deadlocks.

In at least some embodiments, as part of a preliminary phase of thedistributed transaction, the selected coordinator node 1612 may alsoissue read requests directed to the targeted pages, and obtain thecorresponding read logical timestamps (RLTs) for those reads inaccordance with the techniques described earlier. The read logicaltimestamps may be used for making page-level commit decisions at each ofthe storage nodes at which the targeted pages reside, as describedbelow.

The selected coordinator node 1612 may then compose atransaction-prepare (Tx-prepare) message 1642A, which includes anindication of the order in which the targeted pages are to be analyzedfor respective page-level commit decisions, a node chain comprising thestorage nodes responsible for making the page-level commit decisions inthat order, the actual changes to be made to the pages (the bytes to bewritten), and the RLTs for each of the targeted pages. Node chain 1602is shown in FIG. 16 by way of an example. The last or terminal member ofthe node chain (e.g., node 1632C in node chain 1602) may be designatedas a “commit decider” or “decider” node, since its own local page-levelcommit decision may lead to a commit of the transaction as a whole.

The coordinator may transmit the Tx-prepare message 1642A to the firstnode of the node chain, such as storage node 1632A of node chain 1602,which stores at least one of the targeted pages (page P1 of logicalextent E1 in FIG. 16). Node 1632A may perform a local page-level commitanalysis, e.g., using the RLT for page P1 to decide whether the changeto P1 can be committed. Using a technique similar to that describedearlier with respect to conditional writes and RMW sequences, if P1 hasnot been modified since its RLT was obtained, the change to P1 may bedeemed committable. If P1 has been modified since the RLT was obtained,the change may have to be rejected (the rejection scenario isillustrated in FIG. 17 and described below; FIG. 16 illustrates ascenario in which all the page-level commit decisions are affirmative).Assuming that the proposed change to P1 is committable, node 1632A maylock P1 (e.g., acquire a lock managed by a replicated state machine usedfor extent E1) and store an “intent record” in persistent storage. Aslong as page P1 is locked, no reads or updates may be performed on P1 onbehalf of any other transaction or any other RMW sequence in thedepicted embodiment. The intent record may indicate that the node 1632Aintends to perform the proposed modification to P1, and will do so ifthe remaining chain members can also agree to perform their respectivepage-level modifications. Node 1632A may then transmit Tx-preparemessage 1642B (whose contents may be similar or identical to those of1642A) to the next node 1632B of the node chain.

A similar local page-level commit analysis may be performed at node1632B with respect to page P7 of logical extent E5. If node 1632Bdetermines that its local page-level changes are committable (e.g. usingP7's RLT, which was included in the Tx-prepare message 1642B), node1632B may acquire a lock on P7, store its own intent record, andtransmit Tx-prepare message 1642C (similar or identical to 1642B) to thedecider node 1632C.

Decide node 1632C (the terminal or last node in the chain) may performits own page-level commit analysis with respect to page P9 of extent E8.If the proposed modification to page P8 is committable (e.g., if nowrites to P8 have been performed since P8's RLT was obtained by thecoordinator) the decider may determine that the transaction as a wholeis to be committed, and may perform or initiate the proposed changes toP8. The decider node may generate a Tx-commit message 1644A indicatingthat the distributed transaction is to be committed, and transmit it tothe other nodes of the chain. In the depicted embodiment, the Tx-commitsmay be propagated sequentially in the reverse order relative to thepropagation of the Tx-prepare messages. In other embodiments, theTx-commits may be sent in parallel to some or all of the non-decidernodes and/or the coordinator, or may be sent in a different sequentialorder than that shown in FIG. 16.

When a non-decider node of chain 1602 receives the Tx-commit message, itmay perform or initiate its local page-level modifications, release thelock on the local targeted page (e.g., P7 in the case of node 1632B andP1 in the case of node 1632A), delete the intent record it had generatedearlier for the page, and (if required) transmit the Tx-commit messageto another node (e.g., node 1632B may send Tx-commit message 1644B tonode 1632A, and node 1632A may send Tx-commit message 1644C back to thecoordinator). When the coordinator node 1612 receives the Tx-commitmessage, in some embodiments it may transmit a write success response1650 to the requester of the multi-page write 1610. The techniquesdescribed above, of performing local page-level commit analyses in apre-determined order determined to avoid deadlocks, locking pages onlywhen a Tx-prepare message is received and the local commit analysissucceeds, and storing intent records in persistent storage (from whichthey may be accessed in case the storage node responsible for the intentrecord is replaced as a result of a failure that may occur before thetransaction completes, for example), may all help increase theefficiency and recoverability of operations that require atomicity formultiple writes in distributed storage services.

In at least some embodiments, any one of the storage nodes of the nodechain identified for a given distributed transaction may decide, basedon its local commit analysis, that the proposed modification for itslocal page is not acceptable, and may therefore initiate an abort of thetransaction as a whole. FIG. 17 illustrates an example message flow thatmay result in an abort of a distributed transaction at a file storageservice, according to at least some embodiments. As in the case of FIG.16, node 1612 may be selected as coordinator of a distributedtransaction attempted in response to a multi-page write request 1610.The coordinator may perform a preliminary set of operations of thetransaction similar to those described in the context of FIG. 16, suchas determining an order in which local page-level commit decisions areto be made and locks are to be acquired, generating the node chain 1602and creating the Tx-prepare message 1642A. The Tx-prepare message may besent to the first node 1632A of the chain by the coordinator 1612.

Node 1632A may perform its local commit analysis, and decide that theproposed changes to page P1 of extent E1 are acceptable. As in thescenario shown in FIG. 16, node 1632A may acquire a lock on P1, store anintent record in persistent storage, and transmit Tx-prepare message1642B to the next node 1632B of chain 1602. In the scenario illustratedin FIG. 17, node 1632B may decide that the proposed changes to page P7of extent E5 are not acceptable, e.g., because P7 has been successfullymodified since its RLT was obtained by the coordinator 1612.Accordingly, instead of storing an intent record indicating that it iswilling to perform the proposed modification to P7, node 1632B mayinstead generate a Tx-abort message 1744A, indicating that thetransaction should be aborted. The Tx-abort message 1744A may be sent tothe node from which the Tx-prepare message 1642B was received in thedepicted embodiment, although in other embodiments it may be sent inparallel to other node chain members that have already stored intentrecords after successful local commit analyses. Upon receiving theTx-abort message 1744A, node 1632A may delete its intent record, releasethe lock on page P1, and transmit the Tx-commit message 1644C back tothe coordinator 1612. The coordinator 1612 may in turn send a writefailure response 1750 to the requester of the multi-page write in someembodiments. In at least some embodiments, and depending on thesemantics of the APIs being used, neither a write failure response 1750nor a write success response 1650 may be transmitted in at least someembodiments instead, the requesting entities may determine whether theirrequests succeeded or not using other commands (e.g., a directorylisting command may be used to determine whether a delete or renamesucceeded). It is noted that not all the nodes in the node chain mayparticipate in a transaction that gets aborted—e.g., decider node 1632Cin FIG. 17 may not even be made aware that it was to participate in thedistributed transaction. Thus, aborts may not end up wasting anyresources at several of the chain members, which may help reduce theoverall amount of processing associated with distributed transactionscompared to some other techniques.

As noted above, one of the participant storage nodes of a node chainidentified for a transaction may itself be selected as a coordinator ofthe transaction in some embodiments. The coordinator need not be thefirst node of the chain in at least some embodiments, nor may thecoordinator necessarily be the decider node. FIG. 18 illustrates anexample of a distributed transaction participant node chain 1804 thatincludes a node designated as the coordinator of the transaction,according to at least some embodiments. As shown, the node chain 1804comprises storage nodes 1632A, 1632B, 1632K and 1632C, with 1632Adesignated as the first node of the chain and 1632C the terminal anddecider node in the chain. The targeted pages of the transaction thatare to be modified include page P1 of extent E1 at node 1632A, page P7of extent E5 at node 1632B, page P4 of extent E6 at node 1632K, and pageP9 of extent E8 at node 1632C. (Although the examples of FIGS. 16, 17and 18 all show only a single page being modified at each chain member,in general any number of pages may be modified at each chain member invarious embodiments.) Node 1632K has also been designated as thetransaction coordinator.

Accordingly, in its role as transaction coordinator, node 1632K may sendthe Tx-prepare message 1801 to the first node 1632A of the chain. As inthe scenario illustrated in FIG. 16, Tx-prepare messages may bepropagated sequentially along the node chain, e.g., Tx-prepare 1802 maybe sent from node 1632A to node 1632B, Tx-prepare 1803 may be sent fromnode 1632B to 1632K, and Tx-prepare 1804 may be sent from node 1632K tothe decider node 1632C, assuming the respective local page-level commitdecisions at each of the intermediary nodes are positive.

The decider node 1632C may initiate a propagation of Tx-commit messagesin the reverse sequence, e.g., Tx-commit message 1851 may be sent fromnode 1632C to node 1632K, Tx-commit message 1852 may be sent from node1632K to node 1632B, and Tx-commit message 1853 may be sent from node1632B to node 1632B. To complete the transaction, in the depictedembodiment, node 1632A may send a final Tx-commit message 1804 to thecoordinator node 1632K. In at least some embodiments, the dynamicselection of participant nodes of the node chains as coordinators mayhelp to more evenly distribute the coordination workload (e.g., workloadrelated to the preliminary phases of the transaction during which theinformation needed for Tx-prepare messages is collected and analyzed)among the storage subsystem nodes than would have been possible if thecoordinator were chosen statically.

In at least some embodiments, each of the node chain members may storetransaction state records locally for some time even after thetransaction, as discussed below with reference to FIG. 19. The stateinformation may be used, for example, during recovery operations thatmay be needed in the event that one of the participant nodes failsbefore the transaction is completed (either committed or aborted). Overtime, such transaction state information may use up more and more memoryand/or storage space. Accordingly, in order to free up the memory and/orstorage devoted to state information for older transactions, at somepoint after a given transaction is committed or aborted, the coordinatornode 1632K may transmit Tx-cleanup messages 1871, 1872 and 1873 to thenodes of the chain 1804 in the embodiment depicted in FIG. 18. TheTx-cleanup messages may indicate identifiers of the transactions whosestate records should be deleted from the storage nodes. Accordingly, inat least some embodiments, the storage nodes may remove the specifiedtransaction state records upon receiving a Tx-cleanup message. TheTx-cleanup messages may be sent from the coordinator to the storage nodechain members in parallel (as suggested in FIG. 18) or may be propagatedsequentially in various embodiments. The coordinator may decide totransmit Tx-cleanup messages for a given transaction after a tunable orconfigurable time period has elapsed since the transaction was committedor aborted in some embodiments, and the time period may be adjustedbased on various factors such as measurements of the amount ofstorage/memory space used up by old transaction records at variousstorage nodes. Although the coordinator node happens to be a member ofthe node chain 1804 in FIG. 18, Tx-cleanup messages may be sent bycoordinator nodes regardless of whether the coordinator is a member ofthe node chain or not. In some embodiments a single Tx-cleanup messagemay comprise indications of several different transactions whose recordsshould be cleaned up. In at least one embodiment, instead of thecoordinator sending Tx-cleanup messages as shown in FIG. 18, some otherselected member of the chain may be responsible for transmitting theTx-cleanup messages. For example, the Tx-cleanup messages may be sent bythe first member (e.g., node 1632A in FIG. 18) of the chain in one suchembodiment.

In any distributed computing environment, especially large providernetworks in which thousands of commodity computing and/or storagedevices are being used, the possibility of hardware and/or softwarefailures at some subset of the components has to be dealt with whendesigning the services being implemented. FIG. 19 illustrates exampleoperations that may be performed to facilitate distributed transactioncompletion in the event of a failure at one of the nodes of a nodechain, according to at least some embodiments. Three storage nodesstoring 1932A, 1932B and 1932C are shown storing respective replicas1902A, 1902B and 1902C of the same logical extent E1. Initially, replica1902A is designated the master replica, while 1902B and 1902C aredesignated non-master replicas.

The storage node chain generated for any given distributed transactionmay typically comprise storage nodes where the master replicas of theextents involved in the transaction are stored. Such nodes may also bereferred to as “master nodes” or “leader nodes” with respect to thoseextents whose master replicas are stored there. Changes made at a givennode chain member to a physical page may be propagated among the otherreplicas from the master node. Thus, the messages discussed earlier(e.g., Tx-prepare, Tx-commit and Tx-abort) may typically be sent to themaster nodes for the extents involved in the transaction in at leastsome embodiments.

In the depicted embodiment, the master node 1932A may store intentrecords 1915, page locks 1910, and transaction state records 1905 at apersistent shared repository 1980 that is also accessible to otherstorage nodes at which members of E1's replica group are stored. In atleast some embodiments, each node chain member that participates in adistributed transaction message flow (such as nodes 1632A, 1632B and1632C of FIG. 16, and nodes 1632A and 1632B of FIG. 17) may store atransaction record 1905 indicating its local view of the state of thedistributed transaction at the time that a Tx-prepare, Tx-commit, orTx-abort message is sent from the node chain member. For example, if thecommit analysis for the local page modification indicates that themodification is acceptable, and an intent record to modify the localpage is stored, a transaction state record indicating that thetransaction (identified by a unique identifier selected by thecoordinator and included in the Tx-prepare message) is in a PREPAREDstate from the perspective of the node chain member. When a decider nodedetermines that the transaction as a whole is to be committed, it maysave a transaction record with the state set to COMMITTED. When anon-decider node receives a Tx-commit message, the transaction's state(which was previously PREPARED) may be changed to COMMITTED in thedepicted embodiment. When any node of the chain decides to abort thetransaction, a transaction state record with the state set to ABORTEDmay be stored in repository 1980. When any node chain member receives aTx-abort message, the transaction state record may be modified to setthe state to ABORTED. As mentioned above in the discussion regardingTx-cleanup messages, in at least some embodiments transaction staterecords 1905 may be retained at a given storage node for some timeperiod after the messaging associated with the transaction has completedfrom the perspective of that node. This may be done for various purposesin different embodiments—e.g., to aid in recovery from failuresituations resulting from lost messages, for debugging, for auditpurposes, and so on. When a Tx-cleanup message is received for a giventransaction, the transaction state records may be deleted or archived insome embodiments.

The persistent state repository 1980 may be used so that a failover nodemay take over the transaction-related operations if a master node failsbefore the transaction is completed (e.g., before all the Tx-prepare,Tx-Commit, Tx-Abort or messages that the master is responsible forsending for a given transaction are received successfully at theirintended recipients). For example, as indicated by the arrow labeled“1”, master node 1932A (with respect to extent E1) may write atransaction state record 1905, an indication of a page lock 1910, and anintent record 1915) for a given transaction Tx1 for which it received aTx-prepare message in repository 1980 at time T1. Before thecorresponding Tx-commit or Tx-abort message is received, node 1932 mayfail, as indicated by the “X” and the text labeled “2”. In accordancewith a replicated state management protocol, node 1932B may be selectedas the new master node with respect to extent E1 (as indicated by thelabel “3”), e.g., by designating replica 1902B as the new master. Insome embodiments a consensus-based policy may be used to elect the newmaster. The node chain member that would (prior to the failure of node1932A) have transmitted a Tx-commit or Tx-abort to node 1932A, mayinstead find that the master role with respect to extent E1 has beentransferred to node 1932B, and may therefore send the Tx-commit orTx-abort to node 1932B instead. Because the intent record, lock andtransaction state record were all stored in the persistent repository1980, node 1932B may be able to read the required transactioninformation for Tx1 from repository 1980 and easily perform thetransaction-related tasks that would otherwise have been performed bynode 1932A. In at least some embodiments, the persistent repository 1980may be implemented as a component of the replicated state managementsystem used for propagating changes among replicas, associating logicaltimestamps with reads and writes, and so on.

FIG. 20 is a flow diagram illustrating aspects of operations that may beperformed to coordinate a distributed transaction at a file systemstorage service, according to at least some embodiments. As indicated inelement 2001, a file store operation request that involves amodification may be received, e.g., at a metadata node from an accessnode or from another metadata node. An analysis of the request mayreveal whether multiple pages (containing either metadata, data orboth), e.g., at different extents and/or different storage nodes arerequired to fulfill the request. If only a single page is to bemodified, as detected in element 2004, a Read-Modify-Write sequencesimilar to those described earlier may be initiated (element 2007).

If multiple pages need to be modified or written to (as also detected inelement 2004), a distributed transaction may be started by selecting aidentifying a coordinator node (element 2010). A variety of techniquesmay be used to select a coordinator in different embodiments. In atleast one embodiment, one of the participants involved in thetransaction—e.g., a storage node at which a master replica of one of thetargeted pages is stored, or one of the metadata nodes responsible forgenerating and managing the metadata being affected by the transaction,may be selected. In some embodiments, a set of storage subsystem,metadata subsystem or access subsystem nodes may be designated inadvance as coordinator candidates, and a particular node from among thecandidates may be selected.

The coordinator may collect various elements of information needed tocomplete the transaction (element 2013). Such information may include,for example, a list of all the pages that are to be modified and a listof the corresponding write payloads (content of the bytes to be written)may be generated in the depicted embodiment. The coordinator may alsodetermine, e.g., using a deadlock avoidance mechanism, the order inwhich page-level commit analyses should be performed for the transaction(and hence the order in which locks should be acquired). In someembodiments, for example, using the deadlock avoidance mechanism maycomprise sorting the identifiers of the targeted pages using aconsistent sorting methodology that is applied to all distributedtransactions, so that the order in which locks are obtained on any twopages does not change from one transaction to another. The coordinatormay construct the storage node chain for the transaction in the depictedembodiment, for example by identifying the (current) master storagenodes for all the extents whose pages are targeted, and arranging themin the order in which the commit analyses should be performed. In atleast one embodiment, the coordinator may also be responsible forgenerating a unique transaction identifier (e.g., a universally uniqueidentifier or UUID that incorporates a randomly-generated string). Insome embodiments in which read logical timestamps (RLTs) or operationsequence numbers such as those discussed with respect to the conditionalwrite techniques described above are available for I/O operations, thecoordinator may also read all the targeted pages and determine the RLTsassociated with the reads (element 2016). The coordinator may thenconstruct a Tx-prepare message that indicates the node chain, the writepayloads, and the RLTs, and transmit the Tx-prepare message to the firstnode of the chain (element 2019).

At least in some embodiments, the coordinator may then start a timer setto expire after a selected timeout period, and wait for a response toits Tx-prepare message. If no response is received within the timeoutperiod (as detected in element 2023), in some embodiments a response maybe provided to the client that requested the file store operation ofelement 2001 indicating that the result of the operation is unknown(element 2035). In at least one embodiment, a transaction state recoveryoperation may be initiated, e.g., by sending another Tx-prepare messageto the first node of the chain if that node is still accessible, or to areplacement node for that first node if one can be found or configured.

If, within the timeout period, a Tx-commit message is received at thecoordinator (as determined in element 2026), this may indicate that allthe individual page modifications of the transaction have beensuccessfully performed. Accordingly, in some embodiments, thecoordinator may send an indication that the requested operation hassucceeded to the client that requested the operation (element 2029). Inat least one embodiment, Tx-cleanup messages may be sent to the chainnodes, e.g., asynchronously with respect to the receipt of theTx-commit, so that any resources holding transaction state for thecommitted transaction at the node chain members can be released. Asdiscussed earlier, Tx-cleanup messages may be sent either by thecoordinator or by some other selected chain member, such as the firstmember of the chain.

If a Tx-abort message is received at the coordinator (as also detectedin element 2026), the coordinator may in some embodiments optionallysend an indication to the client that the requested operation failed(element 2032). In some embodiments, Tx-cleanup messages may also besent to those chain members who had participated in the abortedtransaction, either by the coordinator or some other member of thechain. Since transactions may be aborted by any of the chain members,only a subset of the members may have stored transaction state recordsbefore the abort occurred, and hence only a subset of the chain membersmay be sent Tx-cleanup messages in some implementations. In otherimplementations, the Tx-cleanup messages may simply be sent to all thenodes of the chain, and those nodes that had not stored any transactionstate for the transaction identified in the Tx-cleanup message mayignore the Tx-cleanup message.

FIG. 21 is a flow diagram illustrating aspects of operations that may beperformed in response to receiving a transaction-prepare (Tx-prepare)message at a node of a storage service, according to at least someembodiments. A member CM of the node chain constructed by thecoordinator, e.g., a node storing a master replica of one of the extentswhose pages are to be modified as part of the transaction, may receive aTx-prepare message from some other node (e.g., typically either from thecoordinator or from some non-decider member of the chain) (element2101). The Tx-prepare message may indicate, in a list of proposed pagemodifications for the transaction, one or more proposed page-levelmodifications to a page P whose parent extent's master replica is storedat CM. CM may determine whether the changes are acceptable/committablefrom its perspective, e.g., by checking in a write log buffer (similarto the buffers shown in FIG. 14) whether page P has been modified sincea read logical timestamp indicated for P in the Tx-prepare message wasobtained. In some cases multiple page level modifications, either to thesame page or to different pages being stored at CM, may be indicated inthe Tx-prepare message, and all such changes may be checked foracceptability.

If the local page-level modifications are committable, as determined inelement 2107, different actions may be taken depending on whether CM isthe decider (the last member of the node chain) or not. If CM is thedecider (as detected in element 2110), the modifications to the localpage or pages may be initiated, and a transaction record indicating thatthe transaction is in COMMITTED state may be stored in persistentstorage in the depicted embodiment (element 2113). The decider node maythen initiate the propagation of Tx-commit messages to the other membersof the node chain (element 2116). The Tx-commit messages may bepropagated sequentially in some embodiments, e.g., in the reverse orderrelative to the sequential order in which the Tx-prepare messages weretransmitted for the same transaction. In other embodiments, theTx-commit messages may be sent in parallel.

If the local page-level modifications are committable and CM is not thedecider node (as also determined in elements 2107 and 2110), in thedepicted embodiment CM may (a) store an intent record (indicating thatif the remaining node chain members also find their local changescommittable, CM intends to perform its local modifications), (b) lockthe targeted local pages of CM (e.g., to prevent any writes to thosepages until the distributed transaction as a whole iscommitted/aborted), and (c) store a transaction state record indicatingthat the transaction is in PREPARED state (element 2119). CM may thensend a Tx-prepare message on to the next node in the chain (element2122).

If the local page-level modifications are not committable (as alsodetected in element 2107), e.g., if the page P has been written to sincethe RLT for P indicated in the Tx-prepare message was obtained, thetransaction as a whole may have to be aborted in order to supportsequential consistency semantics. Accordingly, CM (which may be anon-decider node or a decider node) may store an indication that thetransaction has been aborted (element 2125). In some implementations, atransaction state record indicating the transaction is in ABORTED statemay be stored. In other implementations, a dummy or “no-op” write recordmay be stored in a local write log buffer (similar to buffers 1450 ofFIG. 14). Such a dummy write would have the same effect as the staterecord indicating the ABORTED state. That is, if for some reason (e.g.,as a result of receiving an erroneous or delayed message) an attempt ismade to re-try the transaction at CM, the retry would fail. CM mayinitiate a propagation of a Tx-abort message to the other nodes in thechain that have already sent Tx-prepare messages (if there are any suchnodes) and/or to the coordinator (element 2128).

FIG. 22 is a flow diagram illustrating aspects of operations that may beperformed in response to receiving a transaction-commit (Tx-commit)message at a node of a storage service, according to at least someembodiments. As shown in element 2201, a node chain member CM, indicatedby the transaction coordinator in the Tx-prepare message for thetransaction, may receive a Tx-commit message. The Tx-commit message may(at least under normal operating conditions) typically be received atsome time after CM has performed its local page-level commit analysisand stored a transaction record indicating the transaction is in aPREPARED state. In response to receiving the Tx-commit message, CM mayinitiate the actual modifications to the local targeted pages (element2104) and modify the transaction state record to indicate that thetransaction is now in COMMITTED state. In some embodiments, depending onthe data durability requirements of extent E, multiple extent replicasmay have to be modified before the local page writes can be consideredcompleted. In some such scenarios CM may wait, after initiating the pagemodifications, until enough replicas have been updated before changingthe transaction record.

CM may then release the lock(s) it was holding on the targeted page orpages (element 2207). In at least some embodiments, the intent recordthat CM had stored when responding to the Tx-prepare message for thetransaction may be deleted at this point (element 2210). As notedearlier, in some embodiments, Tx-commit messages may be propagatedsequentially among the chain members in reverse order relative to theTx-prepare messages, while in other embodiments, parallel propagationmay be used, or some combination of sequential and parallel propagationmay be used. If sequential propagation is being used, or if CM candetermine (e.g., based on indications within the Tx-commit message thatit received) that some nodes of the chain have not yet received aTx-commit message, CM may then transmit a Tx-commit message on to aselected node in the chain or to the coordinator (element 2213). In someembodiments duplicate Tx-commit messages may be ignored—e.g., if a givennode or the coordinator receives a Tx-commit message for transaction Tx1and Tx1 is already recorded as having been committed, the new Tx-commitmessage may be disregarded. In some such embodiments, a non-sequentialpropagation mechanism may be used for Tx-commit messages to shorten thetotal time taken to complete the transaction, in which, for example,each node that receives a Tx-commit message may forward Tx-commitmessages to N other nodes of the chain.

FIG. 23 is a flow diagram illustrating aspects of operations that may beperformed in response to receiving a transaction-abort (Tx-abort)message at a node of a storage service, according to at least someembodiments. As shown in element 2301, a Tx-abort message may bereceived at a chain member CM. Just like a Tx-commit message, a Tx-abortmessage may (at least under normal operating conditions) typically bereceived at some time after CM has performed its local page-level commitanalysis and stored a transaction record indicating the transaction isin a PREPARED state.

In response to receiving the Tx-abort message, CM may release thelock(s) it was holding on the targeted page or pages (element 2304). Inat least some embodiments, the intent record that CM had stored whenresponding to the Tx-prepare message for the transaction may be deletedat this point (element 2307). As in the case of Tx-commit messages, indifferent implementations, either sequential, parallel, or hybrid (i.e.some combination of sequential and parallel) propagation may be employedfor Tx-abort messages. In some embodiments, Tx-abort messages may bepropagated sequentially among the chain members in reverse orderrelative to the Tx-prepare messages, for example. If sequentialpropagation is being used, or if CM can determine (e.g., based onindications within the Tx-abort message that it received) that somenodes of the chain that had earlier sent Tx-prepare messages have notyet received a Tx-abort message, CM may then transmit a Tx-abort messageon to a selected node in the chain or to the coordinator (element 2310).In some embodiments, as with duplicate Tx-commit messages, duplicateTx-abort messages may be ignored—e.g., if a given node or thecoordinator receives a Tx-abort message for transaction Tx1 and Tx1 isalready recorded as having been aborted, the new Tx-abort message may bedisregarded. In some such embodiments, a non-sequential propagationmechanism may be used for Tx-abort messages to shorten the total timetaken to abort the transaction, in which, for example, each node thatreceives a Tx-abort message may forward Tx-abort messages to N othernodes of the chain.

On-Demand Page Allocation Using an Extent Oversubscription Model

In many storage systems, performance goals may sometimes potentiallyconflict with space-efficiency goals. For example, in general, keepingthe amount of metadata (such as structures that compriselogical-block-to-physical-page mappings) relatively small relative tothe amount of data being managed may help to speed up various types offile store operations. If metadata grows too large, the cache hit rateat the access nodes' metadata caches may fall, which may result in moreinteractions between the access and metadata subsystems to service thesame number of client requests. Since at least some metadata may bemaintained on a per-logical-block basis, this would suggest that havinglarge logical blocks (e.g., 4 megabyte or 16 megabyte logical blocks)would be better from a performance perspective than having small logicalblocks. However, if physical pages for the entire logical block wereallocated at the time the first write to the logical block is requested,this might result in suboptimal space usage efficiency. For example,consider a scenario where the logical block size is 4 MB (thus, aminimum of 4 MB of physical space would be allocated for any given fileif enough space for an entire logical block is allocated at a time), andthe median amount of data stored in a file within a given directory orfile system is, say, 32 KB. In such a scenario, a large amount ofphysical storage space would be wasted. If logical block sizes were setto close to the median file size, however, this may result in very largeamounts of metadata for large files, thus potentially slowing downoperations not just directed to the large files but to the file storageservice as a whole.

A number of techniques may be used to deal with the tradeoffs betweenspace efficiency and performance in different embodiments. In onetechnique, an oversubscription model may be used for extents, andphysical pages within a given logical block may only be allocated ondemand rather than all at once (i.e., if a logical block size is set toX kilobytes, and the first write to the logical block has a payload ofonly (X−Y) kilobytes, only enough pages to store X−Y kilobytes may beallocated in response to the first write). In another technique,described after the discussion of the oversubscription model, logicalblocks of different sizes may be employed within a given file storeobject, so that the sizes of at least some of the stripes of the objectmay differ from the sizes of other stripes. It is noted that whileextents may be replicated for data durability in various embodiments asdescribed earlier (including in embodiments at which extents areoversubscribed and/or variable logical blocks sizes are used), theextent replication techniques may be considered orthogonal to thelogical-block-to-page mappings, and to extent oversubscription, asdiscussed here. Accordingly, extent replicas may not be discussed indetail herein with respect to oversubscribed extents or with respect tovariable-sized stripes. To simplify the presentation, a logical extentmay be assumed to comprise a single physical extent with respect to mostof the discussion of extent oversubscription management techniques andwith respect to discussions of techniques used for variable-sizedstripes or variable-sized logical blocks.

FIG. 24 illustrates examples of over-subscribed storage extents at adistributed storage service, according to at least some embodiments. Inthe depicted embodiment, logical blocks of a given file store object(such as files 2400A, 2400B, or 2400C) are all of the same size, and allthe physical pages allocated for a given logical block are part of asingle extent. A physical page within a given extent may typically alsobe of the same size as the other physical pages of the extent in thedepicted embodiment. Thus, in one example implementation, an extent maycomprise 16 Gigabytes of 32-KB physical pages, while a logical block maycomprise 4 megabytes. The sizes of the extents, logical blocks and/orphysical pages may be set using respective configuration parameters inat least some embodiments.

As shown, different logical blocks of the same file may at least in somecases be mapped to different extents, and as a result logical blocks maybe considered the equivalent of stripes. File 2400A comprises LB(logical block) 2402A and 2402B. LB 2402A is mapped on-demand to somenumber of physical pages (PPs) 2410A of extent E2434A. Similarly somenumber of physical pages 2410B at extent E2434B are allocated on demandfor LB 2402B. At extent E2434A, some number of pages 2410A are allocatedon demand for LB 2402L of file 2400B as well as LB 2402P of file 2400C.At extent E2434B, some number of pages 2410B are allocated on demand forLB 2420K of file 2400B and for LB 2402Q of file 2400C. The on-demandallocation technique may be implemented as follows in the depictedembodiment: whenever a write request directed to a particular logicalblock is received, the starting offset within the file, and the size ofthe write payload (e.g., the number of bytes to be written or modified)may be used to determine whether any new physical pages are to beallocated, and if so, how many new physical pages need to be allocated.(Some write requests may not need any new pages to be allocated, as theymay be directed to previously-allocated pages.) Only the number of newphysical pages that are required to accommodate the write payload may beallocated, instead of, for example, allocating at one time the entireset of physical pages that could potentially be written as part of thelogical block. Consider the following example: LB 2402A is 4 megabytesin size, and PPs 2410A are 32 KB in size. A first write to LB 2402A,comprising 28 KB of write payload, is received. Prior to this point, nophysical storage has been allocated for LB 2402A in the examplescenario. The storage service makes a determination that only one PP2410A is needed for the first write (since 28 KB can be accommodatedwithin a single 32-KB page). As a result, only one PP 2410A is allocatedwithin extent E2434A, even though the entire 4 MB of LB 2402A mayeventually have to be stored within extent E2434A, since all the pagesof a given logical block have to be allocated from within the sameextent in the depicted embodiment.

In general, in at least some embodiments, it may not be straightforwardto predict what fraction of a logical block is eventually going to bewritten to; some sparse files may contain small regions of data atwidely different logical offsets, for example. In order to improve spaceusage efficiency in the depicted embodiment, extents E2434A and E2434Beach may be oversubscribed. An extent may be considered to beoversubscribed if it is configured to accept write requests to morelogical blocks than could be fully physically accommodated within itscurrent size—e.g., if the complete offset range within all the logicalblocks were somehow to be written to at the same time, the extent mayhave to be enlarged (or a different extent may have to be used). Thus,as shown in oversubscription parameters 2455A, N logical blocks may bemapped to extent E2434A, and each logical block could be mapped to amaximum of M physical pages of Y kilobytes each. Extent E2434A's currentsize is X Kilobytes, where X is less than (N*M*Y). An oversubscriptionfactor OF1 applies to extent E2434A in the depicted embodiment, equal tothe ratio of the potential overflow amount of storage ((N*M*Y)−X) to theactual size of the extent (X). Similar oversubscription parameters 2455Bapply to extent E2434B. E2434B can currently store only up to Zkilobytes, but it is configured to accept write requests directed to Plogical blocks, each of which can be mapped to Q physical pages of R KBeach. Z is less than (P*Q*R), and the oversubscription factor OF2 forE2434B is therefore ((P*Q*R)−Z)/Z. In some embodiments, differentextents may be configured with different oversubscription factors. Inone embodiment, a uniform oversubscription factor may be used for allthe extents. As described below, in some embodiments theoversubscription factor and/or a free space threshold associated withthe oversubscription factor may be modified for at least some extentsover time, e.g., based on collected metrics of file system usage orbehavior. Techniques similar to those described herein foroversubscription management at the per-extent level may also or insteadbe applied to oversubscription at other levels in variousembodiments—e.g., storage subsystem nodes may be oversubscribed based onthe oversubscription of their extents, individual storage devices may beoversubscribed, and so on.

FIG. 25 illustrates interactions among subsystems of a distributedmulti-tenant storage service implementing on-demand physical page-levelallocation and extent oversubscription, according to at least someembodiments. As shown, both metadata extents (such as E2534A) and dataextents (such as E2534B) may be oversubscribed in the depictedembodiment. A first write request directed to a particular logical block(LB) may be received at a metadata node 2522 from an access node 2512,as indicated by arrow 2501. The write request may comprise a writepayload of size “WS”, and may, for example, have been generated at theaccess node 2512 in response to a client's write request directed to afile 2400.

The metadata for the logical block itself may not have been created atthe time the write request 2501 is received—e.g., the write may simplybe the first write directed to a file 2400 after the file is opened. Inthe depicted embodiment, the metadata node 2522 may first generate andwrite LB's metadata. A request 2554 may be sent, for example, to astorage node 2532A to store the LB's metadata. The storage node mayallocate a page from an oversubscribed metadata extent E2534A, and storethe metadata generated by the metadata node 2522, as indicated by block2558. The particular metadata extent to be used may be selected byeither the metadata node 2522, the storage node 2532A, or by a differentplacement component of the storage service in different embodiments. Theselection may be based, for example, on various factors such as the nameof the file being modified, the amount of free space available invarious extents, and so on.

The metadata node 2522 may also determine how many new physical datapages are to be allocated to store the write payload of WS bytes in thedepicted embodiment. A request 2562 for the appropriate number ofphysical pages to accommodate WS bytes may be sent to a differentstorage node 2532B in at least some embodiments than is used for the LBmetadata. The storage node 2532B may allocate the requested number ofphysical pages (which may in at least some cases be less than the numberof pages that would be required if the entire address range of thelogical block were written at once) at an oversubscribed data extent2534B in the depicted embodiment. The identities of the physical pagesmay be stored within the LB metadata stored at extent 2534A in thedepicted embodiment—e.g., the storage node 2534B may transmit theaddresses of the data pages within extent 2534B to metadata node 2522,and metadata node 2522 may submit a request to storage node 2532A towrite the addresses within the LB metadata. In some embodiments, thedata pages may be allocated before the metadata pages are allocated, sothat for example the allocation of the metadata page can be combinedwith the writing of the data page addresses without requiring additionalmessages. In one embodiment, the write payload may be transmitted to thestorage node 2532B by the metadata node 2522 together with theallocation request 2562 for the data pages, in which case the writing ofthe WS bytes may be combined with the allocation of the data pages,without requiring additional messages. In at least some embodiments,after the data page or pages have been allocated for the first writerequest 2501, the identity of the appropriate storage node (2532B) atwhich the data is to be stored may be provided to the access node 2512,and the access node 2512 may submit the write payload to the storagenode.

In at least some embodiments, as mentioned earlier, the use of theoversubscription model may result in situations where a given extent mayrun short of sufficient storage space for all the logical blocks whosecontents it is designated to store. Accordingly, in some embodiments,oversubscribed extents may have to be expanded from time to time, orextent contents may have to be moved or copied from their originalextent to a larger extent. In some embodiments, in order to avoidsynchronous delays that might otherwise result if extent-level datacopying or extent expansion is supported, free space thresholds may beassigned to oversubscribed extent. An asynchronous extent expansionoperation, or asynchronous transfer of extent contents, may beimplemented in such embodiments if the free-space threshold is violated.Different extents may grow at different rates, depending on the natureof the storage workload directed to them. A maximum extent size may bedefined for at least some extents (e.g., based on the capacity of theparticular storage devices being used). As a result, when such a maximumextent size is reached for a particular extent, the extent may no longerbe considered as oversubscribed, and the storage service may employdifferent logic to deal with such maximally-sized extents than the logicused for extents that can still grow. In some embodiments, selectedextents may be moved to a different storage node or a different storagedevice proactively in order to make room for growth of other extents.Such proactive moves may in some implementations be performed asbackground tasks, so as to minimize disruption of ongoingclient-requested operations. A number of different rules, policies orheuristics may be used to select which extents are to be movedproactively to make room for other extents in differentembodiments—e.g., in one embodiment, extents with most of their capacityunused may be chosen for proactive moves in preference to extents withmost of their capacity already in use. The opposite approach may be usedin other embodiments—e.g., extents that have already reached theirmaximum size (or are closer to reaching their maximum size) may be movedin preference to those that still have substantial growth possible.Similarly, the target storage devices or storage nodes to which theextents are moved may also be selected based on configurable policies invarious embodiments. In one embodiment, extents may only be moved whenabsolutely necessary (e.g., proactive moves may not be implemented).

FIG. 26a illustrates an extent for which a free space threshold has beendesignated, while FIG. 26b illustrates an expansion of the extentresulting from a violation of the free space threshold, according to atleast some embodiments. As shown in FIG. 26a , the free space thresholdset for an oversubscribed extent E2634A may be set such that a maximumlimit 2650 of M physical pages may be allocated within the extent beforeexpansion is triggered. As long as the number of allocated pages K ofextent 2634A is less than M (i.e., the number of unallocated pages L isabove the free threshold limit), new pages may be allocated on demand inresponse to write requests as illustrated in FIG. 25. If/when the Mthpage is allocated, an asynchronous copying of the contents of theoriginal extent 2634A to a larger or expanded extent 2634B may beinitiated, as indicated by arrow 2655 of FIG. 26b . As shown, themaximum allocation limit (N pages) of the expanded extent 2634B may belarger than the allocation limit of M pages of the original extent2634A. In some embodiments, it may be possible to expand at least someextents without copying the pages—e.g., if a given oversubscribed extentis located on a storage device with sufficient space to accommodate adesired expansion, the size of the extent may be increased within thestorage device. In other embodiments, the contents of the originalextent may have to be copied to a different storage device, potentiallyat a different storage node. Thus, in one implementation, expandedextent 2634B may occupy a different physical storage device than theoriginal extent 2634A. In at least some implementations, extents ofseveral different sizes may be created at the storage service—e.g., N1extents of 10 GB may be created, N2 extents of 20 GB may be created, andso on. In such embodiments, expansion of an extent may involve copyingpages from a 10 GB extent to a pre-existing 20 GB extent, for example.The term “extent expansion”, as used herein, is intended to refergenerally to any of these types of operations that lead to the abilityto store additional data or metadata contents at an oversubscribedextent when its free space threshold is violated—e.g., whether theoperation involves in-place enlargement of an extent or a transfer ofextent contents from one storage device to another. Thus, an extent mayin some embodiments be expanded by, in effect, replacing the storagedevice being used for the extent with a different storage device, eitherat the same storage node as the original device or at a differentstorage node. In some embodiments, if an extent identifier E1 was usedto refer to the extent prior to the expansion, and a different storagedevice is used post-expansion, a different extent identifier E2 may beused post-expansion. In other embodiments, the same identifier may beused post-expansion.

FIG. 27 is a flow diagram illustrating aspects of operations that may beperformed to implement on-demand physical page allocation at storageservices that support extent oversubscription, according to at leastsome embodiments. As shown in element 2701, a plurality of physicalextents may be set up at a plurality of storage subsystem nodes of adistributed multi-tenant file storage service. In some embodiments, somenumber of extents of one or more different sizes may be pre-configuredat the time that the storage service is started up at a set of resourcesof a provider network, for example. In other embodiments, a set ofextents may be set up when a new file store (e.g., a file system) isinitialized. Each extent may comprise enough space for some selectednumber of physical pages, with each page comprising some number of bytesthat can be used for storing contents of logical blocks of either dataor metadata in some embodiments. For example, in one embodiment, each ofa set of extents may comprise 8 Gigabytes of storage space on aparticular SSD or rotating-disk-based storage device, the defaultlogical block size being used objects whose contents are to be stored atthe extent may be 4 MB, and the physical page size may be set to 32 KB.With this set of parameters, each logical block may comprise up to 128physical pages, and each extent may store up to approximately 2000fully-populated logical blocks (blocks to which at least 4 MB of datahas actually been written, so that there are no unwritten ranges ofoffsets within the logical blocks). In general, it may be the case thatnot all the ranges of offsets within the logical block may contain data(or metadata), since in at least some file system protocols writes maybe directed to random offsets within a file or a metadata structure. Thecontents of a given logical block may be contained within a given extentin the depicted embodiment—e.g., all the physical pages to which thelogical block is mapped may have to be part of the same extent.

Because of the potential for unwritten gaps in the logical blocks, a setof oversubscription parameters may be determined for at least somesubset of extents (element 2704), in accordance with which more logicalblocks may be assigned to a given extent than could be accommodated ifthe blocks were to be fully populated. The parameters for a given extentmay indicate, for example, the oversubscription factor (e.g., a measureof how much additional space could potentially be required for thelogical blocks mapped to the extent), one or more thresholds (such asthe free space threshold discussed above) at which various actions suchas extent expansion are to be triggered, preferred storage devices orextents to which the contents of the current extent should becopied/moved if the thresholds are met, and so on.

In response to a particular write request directed to a logical blockLB1 of a file store object, such as the first write to a file or to ametadata structure, a particular extent E1 of the available extents maybe selected to store contents of the logical block (element 2707). Forexample, E1 may be capable of storing up to P1 pages in all (which couldbe part of several different file store objects in a multi-tenantenvironment), including up to M pages of LB1. In at least some scenariosE1 may be oversubscribed at the time that it is selected—e.g., thecombined sizes of the logical blocks mapped to it (at least some ofwhich may not be fully populated with data or metadata) may exceed thecurrent size of E1. E1 may be selected based on various criteria indifferent embodiments, such as the fraction of its storage space that isfree, the type of storage device (SSD or rotating disk-based) that ispreferred for the file store object, etc. One or more pages may beallocated within E1 for the first write, and the payload of the firstwrite request may be written thereto (element 2710). While the combinedsize of the allocated pages may be sufficient to accommodate thepayload, the combined size of the allocated pages may at least in somecases be smaller than the size of the logical block LB1 (e.g., if thepayload size is smaller than LB1's size). Under normal operatingconditions, in at least some embodiments E1 would only have beenselected for the first write if implementing the write would not violateE1's free space constraints.

A subsequent write request with a write payload of size WS directed toE1 may be received (element 2713). The subsequent write request may bedirected either to LB1 or to some other logical block mapped to E1. Ifallocating enough physical pages to accommodate the write payload WSwould not violate the free space threshold set of E1 (as detected inelement 2716), the required number of physical pages may be allocated,and the requested write may be performed (element 2719). If E1's freespace threshold would be violated (as also detected in element 2716), inthe depicted embodiment one synchronous operation and one asynchronousoperation may be initiated. Synchronously with respect to the writerequest, e.g., so as to avoid any lengthy delays in responding to thewrite request, one or more additional pages would be allocated withinE1. Asynchronously, an extent expansion operation of the kind discussedabove with respect to FIG. 26b may be initiated. The extent expansionmay involve, for example, an in-place enlargement of E1 by changingE1-related metadata at its original storage device, or it may involvetransferring at least some of E1's contents to some other storage device(and/or some other storage node) at which a larger extent may beconfigured. It is noted that in at least some embodiments, E1 may be oneextent replica (such as the master replica) of a replica groupconfigured in accordance with a data durability policy associated with afile store of which LB1 is a block and writes performed at E1 may bepropagated to one or more additional replicas in accordance with thekinds of replication techniques (e.g., erasure coding, full replication,etc.) discussed earlier. At least in some embodiments in which extentsare oversubscribed and pages within a given block are allocatedon-demand, the sizes of pages within a given extent or logical block maydiffer, and/or the sizes of logical blocks within a given file ormetadata structure may differ.

Dynamic on-demand page-level allocation of storage may have the sideeffect of separating parts of the same logical block—e.g., the pagesallocated for a given logical block may at least in some cases not becontiguous on the storage device(s) being used. In some embodiments, itmay be possible to monitor various characteristics of file storeoperations over time, and optimize the way in which extentoversubscription is being implemented, including for example the degreeof oversubscription, as well as the manner in which pages of a givenlogical block are laid out on a physical storage device. FIG. 28 is aflow diagram illustrating aspects of operations that may be performed todynamically modify extent oversubscription parameters, according to atleast some embodiments. As shown in element 2801, physical pages may beallocated over a time period T1 for data and/or metadata in accordancewith an initial set of oversubscription parameters set for some set ofextents E1, E2, etc.

A number of different metrics may be collected during T1 on the filestore operations being performed using the oversubscribed extents(element 2804). For example, file access patterns may be analyzed, e.g.,to determine the proportions of reads and/or writes that are randomversus sequential. Statistics on file sizes (e.g., on the mean or medianfile size, and on how a file's size tends to change over time), on gapswithin files (e.g., the extent to which logical blocks are populated),and/or on response times and throughputs for various types of operationsmay be collected. In some embodiments and for certain types ofoperations, it may be feasible to infer likely patterns of file accessfrom the file names—e.g., file used to store e-mails may be identifiablebased on file name extensions and may be expected to be accessed in aparticular way, files used for database logs or web server logs may beidentifiable by name and may have characteristic access patterns, and soon. Such information and metrics on storage use may be analyzed, e.g.,at optimizer components of the storage service in accordance with amachine learning technique, to determine whether modifying any of theoversubscription parameters may be advisable, or whether the physicalpages of some logical blocks should be consolidated. If a determinationis made that changing oversubscription thresholds may improve spaceutilization levels (element 2807), the threshold may be modifiedaccordingly (element 2810) and a new set of metrics with the modifiedparameters may be collected. For example, in one embodiment,oversubscription parameter settings for a file system FS1 may initiallybe set conservatively—e.g., an oversubscription factor of only 10% maybe set. Later, after storage use metrics and address range gaps forobjects within FS1 are analyzed, the allowed oversubscription level maybe increased, say to 20%. If it is determined that file storeperformance (e.g., for sequential reads/writes) may be improved byrearranging the physical pages of some set of logical blocks, contentsof selected physical pages may be rearranged (element 2813) (e.g., byallocating contiguous space to hold the contents of a given block, andcopying the contents of the block from their original non-contiguouslocations to the contiguous locations). In at least some embodiments,such rearrangements may typically be performed asynchronously withrespect to incoming I/O requests, so that the clients issuing theread/write requests do not experience delays due to the optimizationoperations. Other types of optimizations, such as for example movingsome extents to faster storage devices (such as SSDs) or slower storagedevices than the ones currently being used, may also be initiated on thebasis of similar analysis in various embodiments.

Variable Stripe Sizes

In some embodiments, another approach may be taken to the tradeoffsdiscussed above between metadata size and storage space efficiency. Insome embodiments employing this technique, extents need not beoversubscribed, and all the storage that could potentially be requiredfor a given logical block may be acquired up front, e.g., at the timethat the first write is directed to the block. However, logical blockswithin a given storage object (which, as discussed above, may representthe units of striping file data and/or metadata across extents, storagedevices or storage nodes) may not all be of the same size. In some suchembodiments, the logical block size, and hence the amount of spaceallocated at a time, may be increased as a function of the logicaloffset within the file. Starting with a relatively small amount ofstorage space being allocated for the first block, more and more spacemay be allocated for subsequent blocks; thus, both it may be possible toimplement both small files and large files without creating an amount ofmetadata that increases linearly with object size.

FIG. 29 illustrates examples of file store objects whose contents arestored using variable stripe sizes, according to at least someembodiments. Recall that, as discussed with reference to FIG. 4,different logical blocks of a file store object may typically (althoughnot necessarily) be mapped to different extents at different storagedevices at respective storage nodes, and that logical blocks maytherefore be considered equivalent to stripes. A file 2900 is selectedas an example of a storage object, although various metadata structuresmay also be implemented using variable stripe sizes in variousembodiments. File 2900 is shown as comprising four stripes or logicalblocks LB 2902A, 2902B, 2902C and 2902D. At least some of the logicalblocks 2902 may differ in size from at least some of the others,although some subset of the logical blocks may be of the same size.

Two types of extents are shown in FIG. 29—extents with fixed-size pagesand extents with variable-sizes pages. Extent 2934A comprises physicalpages 2910, each of which is S1 KB in size. Extent 2934B's pages 2910Bare each S2 KB in size, while each of extent 2934C's pages is S3 KB insize. S1, S2 and S3 may differ from each other in the depictedembodiment, e.g., S1 may be smaller than S2, and S2 may be smaller thanS3. As mentioned earlier, at least for extents with fixed page size,physical pages may represent the smallest units of I/O that aresupported in some embodiments. Thus, it may be possible to supportsmaller reads and writes at extent 2934A than at 2934B or 2934C in thedepicted embodiment. Extent 2934D supports variable-size pages—i.e., anarbitrary amount of physical space (with some specified minimum andmaximum) may be allocated at a time within extent 2934D. In contrast,within extents 2934A, 2934B and 2934C, space may be allocated inmultiples of their respective page sizes. In at least some embodiments,only a discrete set of page sizes, or a single page size, may besupported.

In response to the first write directed to an LB 2902, physical storagespace for the entire stripe (which may be more than the physical spacerequired for the write payload of the first write) may be allocated froma selected extent in at least some embodiments. Thus, for example, oneor more pages 2910A of extent 2934A may be used for LB 2902A, and one ormore pages 2910B of extent 2934B may be used for LB 2902B. Similarly,for LB 2902C, one or more pages 2910C may be allocated from extent2934C, and one or more pages from extent 2934D may be allocated for LB2902D. In some embodiments, any given logical block or stripe may bemapped to one contiguous region of physical storage space, while inother embodiments, the physical space allocated for a given logicalblock may be non-contiguous within the storage device address space inat least some cases. If relatively small stripe sizes are used, forexample, for the first few stripes of a file, even small files may bestriped across multiple extents, thus obtaining performance benefits ofstriping which may otherwise not have been achieved had a single largestripe size been used.

In general, in the depicted embodiment, when a write request with aspecified offset and write payload size is received, a decision may bemade (based on the offset and payload size) as to whether the writerequires additional storage space to be allocated. Such a decision maybe made in at least some embodiments at a metadata node of the storageservice. If space does need to be allocated, the amount of (typically,but not necessarily) contiguous physical storage space to be allocatedfor the payload may be determined. In at least some embodiments, thatamount of space allocated may depend on the write offset. (Examples ofstripe sizing patterns over the course of a file's existence, and ofsome of the kinds of factors that may be taken into account whendeciding stripe sizes, are discussed in greater detail below.) One ormore storage nodes may be identified that have extents that can be usedto allocate the desired amount of space. For example, if space for aone-kilobyte stripe is to be allocated, the storage service may attemptto identify extents that have 1 KB pages and have enough free space toaccommodate the write of the stripe. It is noted that the minimum pagesize at a selected extent need not be equal to the stripe or logicalblock size—for example, the stripe size may be 3 KB, but an extent thatsupports 4 KB pages may be used, or another extent that supports 2 KBpages or 1 KB pages may be used. After physical storage for the desiredstripe size is obtained, the modifications indicated in the writepayload may be initiated. In some embodiments in which extents arereplicated, for example, the modifications may be coordinated from thestorage node at which the master replica is stored, and may bepropagated to the non-master replicas from or by the master node.

In some embodiments, stripe sizes within a given file or metadatastructure may change as a function of offset in a predictable fashion.FIG. 30 illustrates examples of stripe sizing sequences that may be usedfor file store objects, according to at least some embodiments. Instripe size sequence 3010A, the sizes of the first nine logical blocksof a file store object may be set, respectively, to 1 KB, 1 KB, 2 KB, 2KB, 4 KB, 4 KB, 8 KB, 16 KB, and 32 KB, for example. Such a pattern maybe used, for example, for files or metadata structures that are expectedto be small, or for files or structures that are expected to growrelatively slowly. For other files, to which for example a large numberof sequential writes are expected with some high probability, adifferent stripe size sequence 3010B may be used, in which the sizes ofthe first four blocks are set to 1 MB, 4 MB, 16 MB and 64 MBrespectively. Thus, even in implementations in which a discrete set ofstripe sizes is implemented, a stripe size used for one file F1 maydiffer from any of the stripe sizes used for a different file F2. Insome embodiments, at least some of the stripe size sequences 3010 to beused may be specified as configuration parameters of the storagesubsystem. In some cases, as a file grows, it may be useful (for bothmetadata performance and for data access performance) to consolidatesmaller stripes into larger stripes.

FIG. 31 illustrates examples of factors that may be taken intoconsideration at a metadata subsystem to make stripe sizing decisions3170 and/or consolidation decisions 3172 for file store objects,according to at least some embodiments. In the depicted embodiment, ametadata subsystem node 122 may be responsible for determiningstripe/logical block sizes for various file store objects, includingfiles and metadata structures, and for determining if and when physicalpages and/or logical blocks should be combined or consolidated. Whendetermining the stripe size to be used for the next portion of a filestore object for which space is to be allocated, the metadata node 112may consider the current size 3101 of the object and the write requestpayload size 3103. In one implementation, for example, the size of thefirst stripe allocated for a file store object may be based on the writepayload of the first write directed to the object—e.g., if the payloadof the first write is 3.5 megabytes, a 4 megabyte stripe size may beselected, while if the first write is less than or equal to 2 megabytes,a 2 megabyte stripe size may be selected. In some embodiments, when afile or directory is created at the request of a customer, hints 3105may be provided to the storage service, indicating for example whetherthe object is going to be used primarily for sequential writes andreads, random writes and reads, or some mix of sequential and randomaccess, and such hints may be used to select stripe/logical block sizes.Metrics 3110 of file system performance, such as the average responsetimes achieved for writes and/or reads of different sizes, may alsoinfluence the selection of logical block size in some embodiments,and/or the scheduling of consolidation operations in which contents ofearlier-created stripes are combined into larger stripes.

In some scenarios, as discussed earlier, the name (or part of the name,such as a file extension) of a file or directory may provide someguidance on the manner in which contents of the file or directory areexpected to grow or be accessed. For example, some applications such ase-mail servers, web servers, database management systems, applicationservers, and the like use well-known file extensions and/or directoryhierarchies for various parts of their functionality, and it may bepossible for an optimizer component of the metadata node 112 to selectstripe sizes more intelligently based such file/directory names 3115. Inat least one embodiment, the metadata node 112 may determine the accesspatterns (e.g., random versus sequential, percent read versus percentwrite, read size distributions, write size distributions) and choosestripe sizes accordingly. Measurements 3125 of object lifetime (e.g.,how much time, on average, elapses between a file's creation anddeletion at a given file store) may be helpful in making stripe sizedecisions in some embodiments—for example, if most files within a givendirectory are expected to be deleted within X hours after creation, thedecisions regarding their stripe sizes may not have much long-termimpact. In some embodiments, extent space utilization metrics 3130and/or storage node resource utilization metrics 3135 (such as CPU,memory, or network utilization levels of the storage nodes being used)may also play a role in determining stripe sizes. In one embodiment,small stripes of a given file or metadata structure may be combined intolarger stripes based on one or more triggering criteria, e.g., if/whenthe file or structure grows beyond a threshold size or if/when frequentsequential accesses to the file are detected. Depending on thecharacteristics of the extents being used (e.g., on the particular pagesizes supported at different extents), such combination operations mayinvolve moving or copying data/metadata from one storage device toanother or from one extent to another. In at least some embodiments, amachine learning technique may be employed to improve the stripe sizingand/or consolidation decisions being made at the storage service overtime. As part of such a machine learning approach, the relative impactof the various factors illustrated in FIG. 31 on overall file storeperformance and/or cost may be analyzed.

FIG. 32 is a flow diagram illustrating aspects of operations that may beperformed to implement striping using variable stripe sizes, accordingto at least some embodiments. A write request indicating a write offsetwithin a file store object, and a write payload, may be received orgenerated (element 3201), e.g., at a metadata node 112 of a distributedmulti-tenant storage service. In some cases, the write request could begenerated at an access node 122 in response to a customer-issued filesystem API call such as a file write, while in other cases the metadatanode may itself decide that some new metadata is to be stored, or thatexisting metadata is to be modified. Based on analysis of the writeoffset, the write payload, and existing metadata (if any) of thetargeted object, a determination may be made that additional storage isto be allocated to implement the write (element 3204). (As mentionedearlier, some writes that consist entirely of modifications ofpre-written content may not require additional storage.)

The size of the next new stripe or logical block of the file storeobject may be determined (element 3207), e.g., based on an offset-basedstripe sizing sequence in use for the file store object (similar to thesequences shown in FIG. 30) and/or on some combination of the factorsshown in FIG. 31, such as the size of the object, the detected accesspatterns, etc. The particular extent, storage node and/or storage deviceto be used to store at least one replica of a stripe of the selectedsize may then be identified (element 3210). As discussed in the contextof FIG. 29, in at least some embodiments, a given extent may beconfigured to use a particular physical page size, and as a result notall extents may be suitable for allocating space for a given logicalblock size; accordingly, the extent may be selected based on the sizesof its pages. In some scenarios, only a discrete set of logical blocksizes that map to a discrete set of physical page sizes of the supportedextents may be permitted. Extents that are configured to supportvariable page sizes (such as extent 2911 of FIG. 29) may be available insome embodiments, and such extents may be selected for allocating spacefor logical blocks/stripes of a variety of sizes. In some embodiments, aplurality of storage nodes (e.g., distributed among several availabilitycontainers or data centers) may be identified for a replica group ofextents when space for a new logical block or stripe is allocated.

An allocation request for the desired amount of physical storage spacemay be sent to at least one selected storage node (element 3213). Thestorage node may allocate the requested physical storage, e.g., enoughpages to store contents of the stripe if the stripe were fully populated(element 3216). The modification indicated in the write request may thenbe initiated or performed (element 3219). Depending on the datadurability policy associated with the file store object, the writepayload may have to be propagated to several different replicas beforethe write can be considered complete. It is noted that at least in someembodiments, on-demand page allocation and/or oversubscribed extents maybe used in combination with variable stripe sizing of the kind describedabove.

Offset-Based Congestion Control Techniques

Customer workloads that access small portions of a data set with highconcurrency can cause hot spots in a distributed file storage service.For example, if a customer requests multiple sequential reads of a fileusing multiple threads of execution at about the same time, all thethreads may end up accessing a single stripe or logical block near thebeginning of the file first. Furthermore, depending on the relativesizes of the logical block and the read payload (the amount of databeing requested in each read request from the customer), multiple readrequests may be directed to a single stripe from each thread. In such ascenario, when many clients request multiple reads from the same logicalblock at about the same time, congestion control techniques may beimplemented within the address range of the logical block to preventpoor overall throughput and/or poor response times for individualthreads. In some embodiments, such congestion control techniques mayassociate offset-based priorities with I/O requests, in which forexample the scheduling priority given to a read request may increasewith the read offset within the logical block.

To motivate the discussion of offset-dependent congestion controltechniques, an illustration of a potentially problematic scenario thatcould result from un-prioritized read request scheduling may be helpful.FIG. 33 illustrates an example timeline of the progress made by multipleconcurrent read requests directed to a logical block of a storageservice object in a scheduling environment in which all the readrequests to the logical block are granted equal priority relative to oneanother, according to at least some embodiments. Extreme values havebeen chosen for various parameters of the example in order to moreclearly illustrate the potential problems; the selected parameters arenot intended as representative of common usage scenarios.

Elapsed time increases from left to right in FIG. 33. At approximatelytime T0, 100 client threads each start a sequential read of a logicalblock 3302 whose contents (e.g., either data or metadata) are stored attwo physical pages PP1 and PP2 of an extent E3334. Logical block 3302may, for example, represent the first logical block of a file which alsoincludes other logical blocks (not shown). Assume that the contents ofLB 3302 are read a page at a time, e.g., to read the entire logicalblock, a given client has to first read PP1 and then read PP2. Theextent E3334 can handle up to 25 page I/Os per second, as indicated byextent capacity 3302. This capacity limit may be assumed to be enforcedin the example scenario illustrated by ensuring that no more than 25page reads are allowed to start during a given second of time. Asindicated by I/O prioritization policy 3301, all the read requests aretreated as having equal priority (which has the same effect as not usingprioritization). Given these parameters, consider the state of theclient requests at the following times along the timeline: T0, T0+1second, T0+2 seconds, T0+3 seconds, and T0+4 seconds.

At approximately T0, 100 requests are waiting to start reading page PP1.Due to the extent capacity constraints, only 25 are allowed to start(and finish) reading PP1 between T0 and T0+1. Accordingly, at T0+1, 75clients are yet to read PP1, while 25 clients have completed readingPP1. However, because all requests are treated with equal priority, itmay well be the case that the 25 clients that have completed reading PP1may not be able to proceed to page PP2 until the remaining 75 clientshave read PP1. Thus, the 25 clients that are indicated by the darkerrounded rectangle at T0+1 may wait for the other 75 to complete readingPP1. At time T0+2, 25 more clients may have completed reading PP1, butthey too may have to wait, until the remaining 50 clients read PP 1. Attime T0+3, 25 clients may have yet to read PP1, and the 75 that haveread PP0 may be forced to wait for them. Only at T0+4, when all 100clients have read the first page, are any of the clients allowed toproceed to page PP2 in the example scenario in which equal prioritiesare assigned to all the read requests directed at the pages of LB 3302.

In at least some embodiments it may be possible to improve overallperformance achieved for the sequential reads by assigning higherpriorities (or, equivalently, lower costs) to those clients that havemade more progress. FIG. 34 illustrates an example timeline of theprogress made by multiple concurrent read requests directed to a logicalblock of a storage service object in a scheduling environment in whichan offset-based congestion control policy is used, according to at leastsome embodiments. Logical block 3302 once again comprises two pages PP1and PP2 at an extent E3334 with a capacity of 25 page I/Os per second.In the depicted embodiment, LB 3302 has an offset-based I/Oprioritization policy 3401 to implement congestion control. Inaccordance with the policy, read requests that are directed to higheroffsets within LB 3302 are given higher priority than read requestsdirected to lower offsets.

At approximately T0, 100 clients begin their sequential read operations.At T0+1, 25 clients have completed reading page PP1, and these 25clients are now requesting reads at a higher offset than the remaining75. According to the offset-based prioritization policy, the 25 clientswho have finished reading PP1 are granted higher priority than theremaining 75 at time T0+1. Thus, those 25 clients now begin reading pagePP2, while the 75 others wait. At time T0+2, the 25 clients havefinished reading all of LB 3302, and can proceed on to the next logicalblock (if any) of the file or metadata structure being readsequentially. Since the next logical block would (with a highprobability) be stored at a different storage device, this means thatstarting from T0+2, the workload of the 100 clients would begin to bedistributed across two storage devices, instead of still being directedto the same extent as in the case where equal priorities were beingused. At T0+3, 25 more clients have finished reading PP1, and aregranted higher priority than the remaining 50 clients that are yet toread PP1. At T0+4, 25 more clients have finished reading both pages, andcan proceed to the next logical block. Meanwhile, 50 clients have yet toread page PP1 at T0+4 in FIG. 34 (which, from the perspective of those50 clients, is a worse outcome than could have been achieved if equalpriorities were being used for all clients as shown in FIG. 33, whereall 100 clients finish reading page PP1 at T0+4). Thus, some clientrequests may be treated somewhat “unfairly” with respect to others inthe scheme illustrated in FIG. 34. As another illustration of theunfairness, consider a scenario in which I/O requests R1 and R2 arereceived at times Tk and (Tk+delta) from clients C1 and C2 respectively,where R1 is directed to an offset O1 within a logical block, R2 isdirected to offset O2 within the logical block, and O2 is greater thanO1. Even though R2 is received after R1, R2 may be assigned a higherpriority based on its higher offset, and hence may be scheduled and/orcompleted earlier than R1 under the scheme of FIG. 34. In some cases, ifR2 is part of a sequential pattern of reads, for example, the entire setof sequential reads may complete as a result of offset-basedprioritization before R1 is scheduled. Despite this “unfairness”,however, the scheme of FIG. 34 would in general tend to lead morequickly to I/O workload parallelism, as the sequential reads of varioussets of clients would tend to get distributed sooner among differentstorage devices than if equal priorities are used for all requestsregardless of offset. In scenarios in which the file store object beingaccessed comprises a plurality of stripes at different store devices(which is expected to be the case for most file store objects), suchspreading of the workload more evenly across storage devices usingoffset-based prioritization may help improve overall average completiontimes and overall throughput for the sequential operations. From theperspective of the components of a multi-tenant storage servicesupporting hundreds or thousands of clients concurrently, it may notalways be straightforward (or efficient) to keep track of whether aparticular page read request is a random read or is part of a sequentialread sequence, and as a result in some embodiments the offset-basedprioritization may be used for page-level reads in general, regardlessof whether the read is part of a larger sequential scan or not. At leastin some embodiments, offset-based prioritization within logical blocksmay be used for any combination of the following types of operations ondata and/or metadata: sequential reads, sequential writes, random reads,or random writes.

A number of different offset-based congestion control techniques basedon similar principles as those illustrated in FIG. 34 may be employed indifferent embodiments. FIG. 35a illustrates an example of a token-basedcongestion control mechanism that may be used for scheduling I/Orequests at a storage service, while FIG. 35b illustrates examples ofoffset-based token cost policies that may be employed, according to atleast some embodiments. Generally speaking, token-based mechanisms maybe used for workload management of various types of entities, such asstorage objects, database tables, database partitions, and the like. Inthe context of a distributed file storage service, such buckets may bemaintained for logical blocks of files, for logical blocks of metadatastructures, for entire files, and/or for entire metadata structures invarious embodiments. A mechanism that uses a single bucket 3508 oftokens 3501 is illustrated in FIG. 35a for simplicity of presentation;however, combinations of multiple buckets may be used in someembodiments, such as one bucket for read operations and a differentbucket for write operations. According to the mechanism, a bucket 3508(e.g., a logical container which may be implemented as a data structurewithin a software congestion control module in at least someembodiments) set up for congestion control purposes associated with aparticular work target such as a logical block of a file may bepopulated with an initial set of tokens 3501 during bucketinitialization, as indicated via arrow 3504A. The initial population maybe determined, e.g., based on expectations of the concurrent workloadlevel, a provisioned operation limit associated with the work target, orsome combination of such factors in various embodiments. For some typesof buckets the initial population may be set to zero in someembodiments. In some implementations the initial population of a bucketmay be set to a maximum population for which the bucket is configured.

When a new I/O request 3522 (such as a read request or a write request)is received, e.g., at a congestion control subcomponent of a storagenode 132, the congestion controller may attempt to determine whethersome number N of tokens (where N may be greater than or equal to 1,depending on implementation or on configuration parameters) are presentin the bucket 3508 in the depicted embodiment. If that number of tokensis available in the bucket, the I/O request 3522 may be accepted forexecution immediately, and the tokens may be consumed or removed fromthe bucket (arrow 3506). Otherwise, if N tokens are not present, theexecution of the requested storage operation may be deferred untilsufficient tokens become available in the depicted embodiment. Thenumber of tokens used up for a given I/O request may be referred to asthe “cost” of accepting the I/O request.

As shown by the arrow labeled 3504B, the bucket 3508 may be refilled orrepopulated over time, e.g., based on configuration parameters such as arefill rate associated with the bucket. In some implementations, tokenrefill operations may accompany, or be performed in close time proximityto, consumption operations—e.g., within a single software routine, Ntokens may be consumed for admitting a request, and M tokens may beadded based on the refill rate and the time elapsed since the bucket waslast refilled. Refill rates or token counts of a given bucket may bemodified in some implementations, e.g., to allow higher work requestrates to be handled, typically for short time intervals. Limits may beplaced on the maximum number of tokens a bucket may hold in someembodiments, and/or on the minimum number of tokens, e.g., usingconfiguration parameters. Using various combinations of configurationparameter settings, fairly sophisticated admission control schemes maybe implemented using token buckets in different embodiments. Inparticular, as described below, by requiring different token costs forI/O requests directed to different offsets, offset-based prioritizationsimilar to the example of FIG. 34 may be implemented.

In one simple example scenario, to support a steady load of 25 I/Orequests per second (IOPS) of equal priority at a logical block LB1,bucket 3508 may be configured with an initial population of 25 tokens, amaximum allowable population of 25 tokens and a minimum of zero tokens.The cost per I/O may be set to 1 token, the refill rate may be set to 25tokens per second, and one token may be added for refill purposes(assuming the maximum population limit is not exceeded) once every 40milliseconds. As I/O requests 3522 arrive, one token may be consumed foreach request. If a steady state workload at 25 IOPS, uniformlydistributed during each second, is applied, the refill rate and theworkload arrival rate may balance each other. Such a steady-stateworkload may be sustained indefinitely in some embodiments, given thebucket parameters listed above. However, if more than 25 I/O requestsare received during a given second, some requests may have to wait, andthe kind of scenario illustrated in FIG. 33 may result.

Instead of setting the cost to 1 token per I/O request, regardless ofoffset, in one embodiment more tokens may be required for I/O requestsdirected towards smaller offsets than are required for I/O requestsdirected towards higher offsets in the file. An example of such a tokencost policy 3576 is shown in FIG. 35b . According to policy 3575, 10tokens are required for each I/O directed to an offset between 0 and 64KB within a logical block, 5 tokens are required for an I/O directed toan offset between 64 KB and 256 KB, and 1 token is required for each I/Odirected to an offset greater than 256 KB. Since more tokens arerequired for lower offsets, I/Os directed to lower offsets may be morelikely to be blocked or delayed for a given token bucket population andrefill rate, while I/Os directed towards higher offsets may in generalbe scheduled more quickly. Various different mathematical functions ormappings may be selected (e.g., based on heuristics, machine learningcomponents of the storage system, or configuration settings chosen by anadministrator) to compute costs from offsets in different embodiments.In some embodiments, a linear offset-based token cost policy 3561 may beused, while in other embodiments non-linear cost policies such as 3562may be used. The cost policies, refill rates and other congestioncontrol parameters being used for various logical blocks, files, ormetadata structures may be modified over time, e.g., in response to theanalysis of performance metrics obtained from the storage service.Different parameters may be used for different logical blocks within agiven file store object in some embodiments, and/or different parametersmay be selected for different file store objects. In at least someembodiments, a similar offset-based congestion control technique may beapplied at the file store object level instead of, or in addition to, atthe logical block level—e.g., a higher priority may be assigned to I/Osdirected to an offset X within a file than is assigned to I/Os directedto an offset Y, where Y is less than X. Instead of using token-basedtechniques, in some implementations, other variable cost assignmenttechniques may be used in some embodiments to assign differentpriorities to I/O requests directed within a logical block or within astorage object. For example, in one embodiment, a numerical cost maysimply be assigned to each I/O request, and outstanding I/O requests maybe handled in inverse order of assigned cost.

In at least one embodiment, respective queues may be set up for I/Orequests directed to different offset ranges within a logical block orfile store object. Each such queue may have an associated delay intervalbefore any one of its queued I/O requests is serviced, with largerdelays assigned to lower-offset I/O requests. FIG. 36 illustrates anexample of the use of offset-based delays for congestion control at astorage service, according to at least some embodiments. In the depictedembodiment, four queues 3602A, 3602B, 3602C and 3602D are shown, eachdesignated for I/O requests (indicated by labels beginning with “R”,such as request R3631) within a particular offset range of a logicalblock. Queue 3602A is used for I/O requests to offsets (e.g., in bytes)between 0 and P−1; queue 3602B is used for I/O requests to offsetsbetween P and 2P−1; queue 3602C is used for I/O requests with offsetsbetween 2P and 4P−1, and queue 3602D is used for I/O requests withoffsets higher than 4P. Each queue 3602 has an associated minimum delay,indicating the minimum time that must elapse between the implementationof any two queued I/O requests of that queue. The minimum delays forqueues 3602A, 3602B, 3602C and 3602D are shown as 4d, 2d, d, and 0respectively. Consider an example scenario in which d is set to onesecond, the population of the various queues at time T is as shown, andno new requests arrive for at least a few seconds. Since requests ofqueue 3602D have a minimum delay of zero seconds, request R3634 may bescheduled first, followed by R3638. Then, requests within queue 3602Cmay be scheduled, with a delay of one second between the completion ofeach request and the commencement of the next. Requests of queue 3602Bmay then be scheduled at two-second intervals, followed by requests ofqueue 3602A with four seconds of delay between each pair of requests. Inthe depicted embodiment, the minimum delays may add to the queuing delayof an I/O request. For example, a particular request R1 may have to waitK seconds in its queue simply because there are other requests in thesame offset range that arrived before R1, and then, when R1 reaches thefront of the queue, R1 may still have to wait for the minimum delayassociated with its queue. The delays between scheduling requests may ingeneral allow higher-offset (and hence higher-priority) requests thatarrive during those delays to be serviced more quickly in the depictedembodiment. A number of variations of the straightforward offset-basedqueuing technique may be used for congestion control in differentembodiments—e.g., in one embodiment, the delay associated a given queue3602 may depend on the number of higher-priority requests that arewaiting for service. In one implementation, a delay to be used for agiven I/O request may be computed simply by multiplying its offset by aconstant.

In some embodiments, error messages may be used as a mechanism forimplementing offset-based prioritization. If a particular I/O request R1is directed to a lower offset some other request or requests, instead ofplacing R1 in a queue or requiring more tokens to be used for R1, anerror message may be returned to the client that submitted R1. Theclient may then retry the I/O (assuming the I/O is still considerednecessary by the client). The delay resulting from the retry may beconsidered analogous to the insertion of the request in an offset-basedqueue as described above.

In at least some embodiments, the storage devices at which the logicalblocks are stored may have to reach a threshold workload level beforethe prioritization policy is enforced. For example, in FIG. 35, theextent E3334 has a defined or baseline capacity of 25 page I/Os persecond, and as a result the prioritization policy may only be appliedwhen the workload exceeds (or at least approaches) the capacity in thedepicted embodiment. The threshold that triggers the prioritization mayitself be a modifiable parameter in at least some embodiments. Forexample, in various embodiments, distinct thresholds may be applied todifferent extents, to different storage nodes, to different physicalstorage devices, or to different logical blocks within the same extent.Such thresholds may be dynamically modified based on various factors. Inone implementation, the threshold could be changed based at least inpart on an overall workload level (e.g., as computed based on astatistical analysis of measurements obtained over some time period) ofthe extent, the storage device or storage node at which the extent islocated, or even the particular logical block to which the threshold isapplied. Other factors that could be used to adjust the thresholds mayinclude, for example, the identity of the client(s) that submit I/Orequests to a logical block or the clients on whose behalf the storageservice object containing the logical block was created (e.g., someclients may be considered more important than others and may thus havehigher thresholds assigned), the time of day (e.g., the threshold may beincreased during typically low-usage periods such as between 11 PM and 6PM), or the names of the file systems, directories, files, volumes orother storage objects implemented using the extent.

In some embodiments, an element of randomness may be added to thecongestion control technique—e.g., instead of implementing fixed delaysfor each offset range, a delay that includes a random component(obtained using some selected random number generator) may be used. Intoken-based congestion control schemes, a random number of tokens may beadded to the requirement for an I/O request to a given offset range.Such randomization may in some cases help to smooth out the workloaddistribution, and may help to reduce the probability of undesirable edgecases in which, for example, storage devices may end up beingunderutilized.

In at least some embodiments, different congestion control policies maybe used for different categories of storage operations. FIG. 37illustrates examples of congestion control policies that may bedependent on the type of storage object being accessed and variouscharacteristics of the requested accesses, according to at least someembodiments. As shown in table 3780, congestion control parametersettings 3710 may vary based on the content type 3702 (e.g., metadataversus data), whether a request is a read or a write (I/O type column3704), and/or on whether the request is part of a sequential or randomsequence (access pattern column 3706). Different congestion controlsettings may also or instead be used based on I/O payload size (column3708) (e.g., how many bytes of data/metadata are being read or written)and/or on the current size of the targeted object (column 3710).

For sequential reads of metadata structures, where the individual readpayload sizes are less than 4 KB and the metadata structure is smallerthan S1 MB, linear offset-based prioritization may be used forcongestion control in the depicted embodiment. Random metadata writeoperations of any size are to be treated as having equal priorities.Sequential data reads with payload sizes greater than 64 KB, directed atfiles with size>128 MB, are to use offset-based priorities withexponential decay as a function of offset. Various details (such as thecost associated with each priority level, the offset boundaries fordifferent priorities, or the minimum delays between requests) of thecongestion control policies have been omitted from FIG. 36 to simplifythe presentation. It is noted that other factors than those shown inFIG. 36 may be used to assign congestion control parameters in differentembodiments, and that not all the factors shown in FIG. 36 need beconsidered in at least some embodiments. For example, in someembodiments, congestion control techniques may only be used forconcurrent sequential reads.

FIG. 38 is a flow diagram illustrating aspects of operations that may beperformed to implement offset-based congestion control for schedulingI/O operations at a storage service, according to at least someembodiments. As shown in element 3801, an I/O request (a read or awrite) directed to at least a portion of a logical block LB 1 of astorage object (such as a file or a metadata structure) being managed bya multi-tenant file storage service may be received. In differentembodiments, offset-based congestion control decisions may be made atany of the various subsystems described above, or by a combination ofsubsystems. In some embodiments congestion control decisions for filereads/writes may be made at access subsystem nodes, while the decisionsfor metadata may be made at the metadata subsystem. In otherembodiments, congestion control decisions may be made at storagesubsystem nodes for both data and metadata. The offset within thelogical block LB 1 at which one or more storage operations are to beperformed to fulfill the I/O request may be determined (element 3804).

Based at least in part on the offset, values of one or more congestioncontrol parameters (e.g., the cost value assigned to the IO request,such as the number of tokens to be consumed from a token bucket, or thedelay before the execution of a storage operation) may be determined(element 3807). In at least some embodiments, the parameters selectedmay favor, or give a higher priority to, requests at higher offsetswithin the logical block LB 1 than to requests at lower offsets. Thestorage operations corresponding to the I/O request may then bescheduled in accordance with the selected congestion control parameters(element 3810). In at least some embodiments and for certain types ofI/O requests, a response may be provided to the requester (element3813). It is noted that the offset-based congestion control techniquessimilar to those described herein may be used in a variety of storageservice environments in different embodiments, including services thatimplement file system interfaces/protocols, services that implement aweb services interface in which the storage object is associated with auniversal record identifier (URI), or services that implement ablock-level device interface.

Consistent Object Renaming Techniques

At a distributed file storage service, object rename operations—e.g.,operations performed in response to customer requests to change the nameof a file or a directory—may at least in some cases involve updates tometadata elements stored at more than one metadata node (or more thanone storage node, if the metadata subsystem stores its structures at thestorage subsystem). Although the distributed transaction techniquedescribed earlier may be used to implement such multi-node renames, inat least some embodiment a different rename-specific mechanism may beused as described below. FIG. 39 illustrates an example of the metadatachanges that may have to be performed at a plurality of metadatasubsystem nodes of a file storage service to implement a renameoperation, according to at least some embodiments. By way of example,the metadata changes needed to rename a file “A.txt” to “B.txt” areillustrated, although similar changes may be made for directory renames,link renames, and the like. In the depicted embodiment, three metadatasubsystem nodes 3922A, 3922K and 3922P of the storage service are shown.Various attributes 3912 of a particular file store object initiallynamed “A.txt”, including for example an identification of the physicalpages being used for the object at one or more storage nodes, a useridentifier and/or a group identifier of the object's owner, the currentsize of the object, the last modification time, the access permissionsor ACLs (access control lists), a link count indicating how many hardlinks point to the object, and so on, may be stored in a DFS node entrystructure labeled DFS-Inode 3910 at metadata node 3922A. The DFS-Inodestructure 3910 may be similar in concept to the inode structuresimplemented in many traditional file systems, with some set of added ormodified features of the DFS-Inode resulting from the distributed natureof the file storage service.

The name “A.txt” of the file store object (prior to the implementationof the rename operation workflow) may be stored in a different metadatastructure called DFS-DirectoryEntry 3930, at a different metadata node3922K in the depicted embodiment. DFS-DirectoryEntry 3930 may include afield 3934 for the object name and a pointer to the DFS-Inode 3910 thatstores the attributes of the object. In at least some embodiments, theDFS-DirectoryEntry 3930 may also include a parent directory pointerDFS-ParentDirPtr 3952 pointing to the DFS-DirectoryEntry of the parentdirectory of the object. Thus, for example, if “A.txt” is in a directory“dir1”, the DFS-ParentDirPtr may point to the DFS-DirectoryEntry of“dir1”. DFS-DirectoryEntry metadata structures may be referred to in thesubsequent discussion simply as directory entries, while DFS-Inodestructures may be referred to simply as node entries.

The particular metadata node 3922A that is chosen to manage a givenobject's directory entry may be selected using different techniques indifferent embodiments, such as by hashing the name of the object at thetime the object is created, by selecting the metadata node based on itscurrent available workload capacity or space availability, and so on. Asa result, a different metadata node 3922P may at least in some cases beselected to manage the directory entry to be created for the secondoperand (“B.txt”) of the “rename A.txt B.txt” operation.

The changes required to implement the rename of “A.txt” to “B.txt” areindicated in FIG. 39 by the labels “Pre-rename state 3945” and“Post-rename state 3947”. To implement the rename workflow, a newdirectory entry 3931 with object name field 3938 set to “B.txt”, and apointer field pointing to DFS-Inode 3910 should be created, and theoriginal directory entry 3930 with the name field “A.txt” should beremoved. The node entry 3910 itself may not be modified during therename in at least some embodiments. For consistency, the combination ofmetadata changes shown in FIG. 39 may have to be performed in such a waythat either all the changes (at both metadata nodes involved) succeed,or none succeed. In some embodiments, as described earlier, the metadatastructures may actually be stored using extents implemented at physicalstorage devices of storage subsystem nodes of the service. In the latterscenario, four types of entities may be involved in a rename workflow,any one of which may fail independently of the other, or mayindependently lose incoming or outgoing network packets: the metadatanode and the storage node of the original directory entry (“A.txt”'sdirectory entry) and the metadata node and storage node of the newdirectory entry (“B.txt”'s directory entry). Accordingly, a renameworkflow designed to take possible failures and/or communication delaysat any of the participant nodes may be implemented, using a sequence ofat least two atomic operations as described below. Each atomic operationof the sequence may be confined to one of the metadata nodes, and maytherefore be easier to implement than multi-node atomic operations. Itis noted that each metadata node (and/or storage node) involved may beconfigured to manage metadata for numerous file store objects,potentially belonging to numerous clients of the storage service in amulti-tenant environment, and as a consequence each metadata or storagenode may have to handle large numbers of rename requests and other filestore operation requests concurrently.

To prevent inconsistency and/or metadata corruption, metadata structuressuch as directory entries may be locked (e.g., using exclusive locks)during rename workflows in some embodiments. In order to preventdeadlocks (as might potentially occur if, for example, two renamerequests “rename A.txt B.txt” and “rename B.txt A. txt” are received invery close time proximity), a lock ordering protocol may be employed inat least some embodiments. FIG. 40 illustrates a use of such a deadlockavoidance mechanism for concurrent rename operations, according to atleast some embodiments. A deadlock avoidance analyzer module 4004 (e.g.,a subcomponent of the metadata subsystem) may take as input the operands4001 of the rename request (e.g., operands “X” and “Y” of a “rename X toY” request) and generate a particular lock acquisition order in thedepicted embodiment.

Two alternative lock acquisition sequences 4010 and 4012, of whichexactly one may be generated as output by the deadlock avoidanceanalyzer module 4004, are shown with respect to a “rename X to Y”request in the depicted embodiment. According to acquisition sequence4010, a lock on X's directory entry is to be obtained as part of a firstatomic operation of a rename workflow. According to acquisition sequence4012, a directory entry for Y is to be obtained (after creating thedirectory entry if necessary) in a first atomic operation of the renameworkflow. In the depicted embodiment, a name comparator 4008 may be usedby the deadlock avoidance module to arrive at the lock sequence. The twooperands may be compared, e.g., lexicographically, and in at least someembodiments the operand that is first in the lexicographic order may beselected as the one to be locked in the first atomic operation. (Inother embodiments, the operand that is second in lexicographic order maybe locked first; as long as the ordering logic is applied consistentlyacross different rename operations, which specific one of the operandsis locked first may not matter.) Thus, in such embodiments, the samedirectory entry may be locked first regardless of whether the renamerequest was “rename X to Y” or “rename Y to X”. In this way, even if tworequests “rename X to Y” and “rename Y to X” are receivednear-concurrently, deadlocks may be avoided, since it would not bepossible for X to be locked for the first request and Y to be locked forthe second request. In some embodiments, techniques other thanlexicographic comparison may be used to determine lock order among therename operands. Since multiple objects (e.g., multiple files ordirectories) may have the same name within a given file store, while theidentifiers assigned to DFS-Inodes may typically be expected to beunique within a file store, in at least some embodiments the “names”used as inputs to the comparator may be obtained by concatenating orotherwise combining the identifier of a selected DFS-Inode associatedwith the object (e.g., the parent DFS-Inode of the object) with theobject's name. Other disambiguation techniques may be used in otherembodiments to overcome potential problems of file name (or directoryname) re-use—e.g., the entire path from the root of the file store tothe object may be used as the “name” for lock sequence determination inone embodiment, or DFS-Inode identifiers associated with several of thepath's directories may be combined with the object name.

In at least some embodiments, based on the output of the deadlockavoidance analysis, one of two different rename workflows may beimplemented for a given rename request. The two workflows may differ inwhich directory entry is locked first. Each of the rename workflows maybe considered as comprising at least three phases: a first set ofoperations performed atomically (which may collectively be referred toas “the first atomic operation” of the workflow), a second set ofoperations performed atomically (which may collectively be referred toas “the second atomic operation”), and a third set of operations forwhich atomicity may be implementation-dependent. Additional (typicallyasynchronous) phases may also be included in some cases as describedbelow. FIG. 41 is a flow diagram illustrating aspects of operations thatmay be performed to implement a first rename workflow based on a firstlock ordering, among two possible lock orderings, that may be determinedat a storage service for a rename operation, according to at least someembodiments. As shown in element 4101, a request to rename a particularfile store object, such as a file or a directory, whose current name is“A” to “B” may be received, e.g., at a metadata subsystem of adistributed storage service. For example, an access subsystem node mayreceive a rename command from a customer, and transmit a correspondinginternal rename request to a selected metadata node. In embodiments inwhich a storage subsystem of the service is used for both metadata anddata, the metadata node may for example comprise a process or threadco-located at the same hardware server as a storage node. A directoryentry for “A” may currently point to a node entry DI1 that comprisesvalues of various attributes of the object, such as ownershipidentification, read/write permissions, and the like. A directory entryfor “B” may not yet exist.

A determination may be made, e.g., based on deadlock avoidance analysis,whether a lock on “A”'s directory entry is to be acquired first as partof the rename workflow, or whether a lock on a directory entry for “B”(which may first have to be created) is to be acquired first (element4104). If B's directory entry is to be locked first (element 4107), theworkflow steps illustrated in FIG. 42 may be used, as indicated by thelabel “Go to 4201” in FIG. 41. If “A”'s entry is to be locked first (asalso determined in element 4107), a first atomic operation of the renameworkflow may be attempted at a particular metadata node MN1 of thestorage service (element 4110). The first atomic operation may comprisethe following steps in the depicted embodiment: (a) obtaining a lock L1on “A”'s directory entry; (b) generating a unique rename workflowidentifier WFID1 for the workflow being attempted and (c) storing anintent record IR1 indicating that the object currently named A is to berenamed to B. In at least some implementations the intent record mayinclude or indicate the workflow identifier WFID1. In oneimplementation, a state management subcomponent of the storage service(e.g., similar to the replicated state machine illustrated in FIG. 12)may be used to combine the three steps into one atomic operation. Theorder in which the three steps of the first atomic operation areperformed relative to each other may vary in different implementations.In some embodiments, respective representations of the lock L1, theintent record IR1 and/or the workflow identifier WFID1 may each bereplicated on persistent storage devices, e.g., using extent replicas ofthe storage subsystem as described earlier. In at least one embodiment,the persistent storage locations selected for storing the lock, theintent record and/or the workflow identifier may be accessible fromreplacement metadata nodes in the event of a failure of MN1. As long asthe lock L1 is held, no other modification may be applied to “A”'sdirectory entry in the depicted embodiment. If the lock is already heldwhen the first atomic operation is attempted, e.g., on behalf of someother concurrent or near-concurrent modification operation, the firstatomic operation may be delayed until the lock becomes available.

If the initial atomic operation succeeds, as determined in element 4113,the second atomic operation of the rename workflow may be attempted. Itis noted that with respect to each of the atomic operations of theworkflows illustrated in FIGS. 41 and 42, in at least some embodimentsthe atomic operation may be re-tried one or more times (e.g., based onsome configurable maximum retry count) in the event that the operationcannot be completed on the first attempt. The second atomic operationmay be performed at the metadata node (MN2) that is designated to manageand/or store the directory entry for “B”. In some embodiments, after thefirst atomic operation is completed at MN1, a request to perform thesecond atomic operation may be sent from MN1 to MN2. The request mayinclude the workflow identifier WFID1 in at least some implementations.As shown in element 4116, the second atomic operation may comprise thefollowing steps: (a) verifying that “B”'s directory entry is notcurrently locked on behalf of some other modification operation (b)setting B's directory entry to point to the node entry DI1 for theobject being renamed and (c) storing a record indicating that, for theworkflow with identifier WFID1, the pointer modification step of “B”'sdirectory entry succeeded. In at least some cases, “B”'s directory entrymay not exist at the time that the second atomic operation is attempted,in which case the step of verifying that it is not locked may beimplemented implicitly by creating a new directory entry for “B”. In atleast some embodiments, a lock may be acquired on B's directory entrybefore the pointer is modified, e.g., to prevent any concurrentmodifications of “B”'s directory entry. The lock may be released afterthe pointer to DI1 is set in some such embodiments. As in the case ofthe writes performed as part of the first atomic operation, the writesof the second atomic operation (e.g., the setting of the pointer and thesuccess indication) may be performed at persistent storage locationssuch as replicated extents from which they may be read later in theevent of a failure at MN2. A state management subcomponent of thestorage service may be used to enforce atomicity of the combination ofthe writes.

If the second atomic operation succeeds (as determined in element 4119),a third set of operations may be attempted (element 4122). Like thefirst atomic operation, this third set of operations may also beexecuted at MN1. In at least some embodiments, an indication received atMN1 that the second atomic operation succeeded (e.g., a response to arequest sent from MN1 to MN2 for the second atomic operation) maytrigger the third set of operations. In the third set of operations, thelock L1 acquired on “A”'s directory entry may be deleted, the intentrecord IR1 may be deleted, and “A”'s directory entry itself may bedeleted. As mentioned earlier, in some implementations, this third setof operations may also be performed as an atomic unit, and in such casesthe operations of the third set may be referred to as the “third atomicoperation” of the workflow. In other implementations atomicity may notbe enforced for the third set of operations. In embodiments in which themetadata generated during the first atomic operation (e.g., the intentrecord, the workflow identifier and the indication of the lock) arestored in persistent storage, the third set of operations may beexpected to succeed eventually, even if one or more retries are requireddue to failures of various kinds, regardless of whether the third set isperformed atomically or not. If the third set of operations succeeds aswell (as detected in element 4125), the rename workflow as a whole maybe deemed to have succeeded (element 4128). In at least some embodimentsa response to the rename request may be sent, indicating that the renamesucceeded. In some embodiments no response may be sent, and therequester.

In the depicted embodiment, if either of the two atomic operations didnot succeed, the workflow as a whole may be aborted (element 4131), andany of the records generated in earlier parts of the workflow may bedeleted (such as the intent record IR1, a representation of theacquisition of lock L1 and/or the success record stored at MN2). If anyoperation of the third set of operations fails as detected in element4125), it may simply be retried in the depicted embodiment as indicatedby the arrow leading back to element 4122. As mentioned earlier, in atleast some embodiment multiple attempts may be tried for each of theatomic operations before declaring failure. In some embodiments, at somepoint after the third set of operations of a workflow with identifierWFID1 is complete, the success record stored at MN2 may be deleted(element 4134), e.g., asynchronously with respect to the completion ofthe third set of operations.

As indicated in the negative output of element 4107 of FIG. 41, adifferent rename workflow may be attempted if the directory entry for“B” is to be locked first. FIG. 42 is a flow diagram illustratingaspects of operations that may be performed to implement a second renameworkflow based on such a second lock ordering, among the two possiblelock orderings, that may be determined at a storage service for a renameoperation, according to at least some embodiments. This second workflowmay also comprise two successive atomic operations to be used to rename“A” to “B” in the depicted embodiment, followed by a third set ofoperations that may or may not be implemented atomically depending onthe implementation. The first atomic operation (element 4201 of FIG.42), performed at the metadata node MN2 (the node responsible forstoring a directory entry for object name “B”) may include verifyingthat “B”'s directory entry is not locked for some other operation,creating “B”'s directory entry if needed, locking “B”'s directory entry,generating and storing a unique workflow identifier WFID2 for the renameworkflow, and storing an intent record IR2 indicating that the objectcurrently named “A” is going to be renamed to “B”. In someimplementations the intent record IR2 may include or indicate theworkflow identifier WFID2.

If the first atomic operation succeeds (as detected in element 4204), asecond atomic operation of workflow WFID2 may be attempted (element4207). This second atomic operation may be performed at the metadatanode MN1 at which “A”'s directory entry is managed, and in someembodiments may be triggered by a request from MN2 indicating that thefirst atomic operation has succeeded. The second atomic operation mayinclude verifying that A's directory entry is not locked, deleting “A”'sdirectory entry, and storing a persistent record that “A”'s directorentry has been successfully deleted as part of workflow WFID2. If thesecond atomic operation succeeds (as determined in element 4210), thethird set of operations may be attempted at MN2 (element 4213). In someembodiments, an indication that the second atomic operation succeeded,e.g., a response received at MN2 to a request sent from MN2 to MN1earlier for the second atomic operation, may trigger the attempt toperform the third set of operations. The third set of operations mayinclude setting “B”s directory entry to point to DI1 (the node entry forthe object being renamed), releasing/deleting lock L2, and deleting theintent record IR2.

If the third set of operations succeeds (as detected in element 4216),the workflow as a whole may be deemed to have succeeded (element 4219),and in some embodiments a success indicator may be returned to therequester of the rename operation. As in the workflow illustrated inFIG. 41, the third set of operations of FIG. 42 may be expected tosucceed eventually, although one or more retries may be required infailure scenarios as indicated by the arrow leading back from element4216 to element 4213. Asynchronously with respect to the completion ofthe third set of operations, the success record stored by MN1(indicating that “A”'s directory entry has been deleted) may itself bedeleted (element 4225) in at least some embodiments. If either of thetwo atomic operations fail, the rename workflow as a whole may beaborted (element 4222), and records stored during earlier operations ofthe aborted workflow may be cleaned up. As in the operations illustratedin FIG. 41, the storage service's state management mechanisms and/orreplicated extents may be used for the atomic operations of the secondworkflow.

Using the deadlock-avoiding lock ordering sequence and the operationsillustrated in FIG. 41 and FIG. 42, rename operations for file storeobjects may be implemented to achieve the desired level of consistencyexpected by the file system protocols being used. The techniques ofstoring intent records associate with unique workflow identifiers inpersistent storage may be helpful in recovery from various types offailures in different embodiments. FIG. 43 is a flow diagramillustrating aspects of recovery operations that may be performed inresponse to a failure of one metadata subsystem node of a pair ofmetadata subsystem nodes participating in a rename workflow, accordingto at least some embodiments, while FIG. 44 is a flow diagramillustrating aspects of recovery operations that may be performed inresponse to a failure of the other metadata subsystem node of the pairof metadata subsystem nodes participating in the rename workflow,according to at least some embodiments. To simplify the presentation,FIG. 43 and FIG. 44 each illustrate operations that may be performed ifa single metadata node failure occurs during the workflow sequenceillustrated in FIG. 41, although similar recovery strategies may beemployed even if both metadata nodes involved in the workflow fail in atleast some embodiments.

As shown in element 4301 of FIG. 43, a failure of node MN1 may bedetected at some point after the first atomic operation (whose stepswere illustrated in element 4110) of FIG. 41's workflow sequencecompletes, and before the third set of operations (element 4122) of FIG.41's workflow sequence is begun. For example, the processes or threadsimplementing the metadata node MN1 where “A” s directory entry ismanaged may exit prematurely, or MN1 may become unresponsive to healthchecks due to a network-related failure or due to a software bug thatresults in a hang. Under such circumstances, a replacement metadata nodeMN-R may be configured or designated to take over the responsibilitiesof MN1 (element 4304) in the depicted embodiment. In some embodiments,as mentioned earlier, MN1 may have been configured as a member of aredundancy group comprising a plurality of metadata nodes, and anothermember of the redundancy group that was preconfigured for failover maybe quickly designated as a replacement. In other embodiments,replacement metadata node MN-R may not be part of a preconfiguredredundancy group.

In the first atomic operation of the workflow of FIG. 41, MN-1 storedintent record IR1 and workflow identifier WFID1 in persistent storage,together with a representation of the lock L1. The replacement metadatanode MN-R may read the intent record IR1 and workflow identifier WFID1that were written prior to MN-1's failure (element 4307). MN-R may thensend a query to MN2, the metadata node responsible for “B”'s directoryentry, to determine the status of the workflow WFID1 (element 4310) inthe depicted embodiment—e.g., to find out whether B's directory entrypointer has already been set to point to DI1 (the node entry of theobject being renamed) as part of the second atomic operation of theworkflow.

As mentioned earlier, each metadata node may be responsible for managingmetadata for several different files and/or for several differentclients in embodiments in which the distributed storage service ismulti-tenant. Consequently MN2 may have stored respective successrecords corresponding to the second atomic operation of numerous renameworkflows. Upon receiving the query regarding the status of the workflowwith identifier WFID1, MN2 may look up its records of successful atomicoperations. If MN2 finds a success record for WFID1's second atomicoperation (as determined in element 4313), it may inform MN-R that thesecond atomic operation was completed (i.e., that “B”'s directory entrywas set to point to the node entry DI1). Accordingly, in the depictedembodiment, MN-R may then attempt the third set of operations in aneffort to complete the rename workflow identified by WFID1 (element4316).

At least in some scenarios, it may be the case that the second atomicoperation of workflow WFID1 does not succeed. For example, MN1 may havefailed before its request to MN2 to start the second atomic operationwas successfully transmitted, or the request may have been lost, or MN2may not have been able to successfully implement the requested secondatomic operation. In some embodiments, if MN-R is informed that thesecond atomic operation had not succeeded (as also determined in element4313), MN-R may have the option of either abandoning or resuming theworkflow. In the depicted embodiment, if a cancellation criterion is met(as detected in element 4319), the rename workflow may be aborted andmetadata record associated with WFID1 that were stored by MN1 may beremoved (e.g., the intent record IR1 and the representation of the lockL1 may be deleted from persistent storage) (element 4322). In oneembodiment, the cancellation criterion may be met if the time that haselapsed since the original rename request was received from a clientexceeds some configured threshold. An elapsed-time-dependent terminationof the rename workflow may be implemented, for example, under theassumption that in view of the long elapsed time, the client thatrequested the rename would have realized that the original request didnot succeed, and would therefore not be expecting the rename to succeedat this point. In some embodiments, a cancellation record indicatingthat the workflow with identifier WFID1 has been aborted/cancelled maybe stored for some configurable time period, e.g., at either MN-R, atMN2, or at both MN-R and MN2. In one such embodiment, after determiningthat the workflow is to be abandoned, MN-R may first send a request toMN2 to store the cancellation record, and may delete both the intentrecord and the lock after it is informed that MN2 has successfullystored the cancellation record to persistent storage.

If, however, the cancellation criterion is not met (as also detected inelement 4319), in the depicted embodiment MN-R may resume the workflowby sending a request to MN2 to implement the second atomic operation(element 4325). Other strategies to respond to MN1 failures may beimplemented in various embodiments—e.g., in some embodiments the renameworkflow may always be resumed regardless of the time that has elapsedsince the initial rename request was received, and in at least oneembodiment the rename workflow may always be abandoned in the event of afailure of MN1 after the completion of the first atomic operation.

FIG. 44 illustrates operations that may be performed if metadata nodeMN2 fails during the workflow sequence illustrated in FIG. 41, accordingto at least some embodiments. As shown in element 4401, a failure of MN2may be detected, for example after a request to implement the secondatomic operation (element 4116) of the workflow is sent to MN2. In amanner similar to that discussed for replacing MN1 by MN-R above, areplacement metadata node MN-R2 may be designated or configured for MN-Rin the depicted embodiment (element 4404). MN-R2 may be able to read thesuccess records written to persistent storage by MN2 prior to itsfailure.

At MN-R2, a query from MN1 may be received to enable MN1 to determinewhether the second atomic operation of the workflow with identifierWFID1 was successfully completed (element 4407). If the second atomicoperation had been completed prior to MN2's failure (as detected inelement 4410), MN-R2 may be able to find a success record for WFID1, andmay respond to MN1 accordingly. MN1 may then resume the workflow byattempting the third set of operations (element 4413).

If the second atomic operation of WFID1 had not been completed, asimilar procedure may be implemented in the embodiment depicted in FIG.44 as was implemented in FIG. 43. If a cancellation criterion for therename operation is met (as detected in element 4416)—e.g., if the timeelapsed since the rename was requested exceeds some threshold time T—therename operation may be aborted and the data structures related to WFID1may be cleaned up (element 4419). Otherwise, if the cancellationcriterion has not been met, the workflow may be resumed by MN1 bysending a request to perform the second atomic operation to MN-R2(element 4422).

While FIG. 43 and FIG. 44 illustrate recovery techniques responsive tofailures at either metadata node during the workflow of FIG. 41,analogous techniques may also be implemented if either metadata nodefails during the workflow illustrated in FIG. 42 in at least someembodiments. As long as the replacement node configured for the failedmetadata node is able to read the workflow records (e.g., the intentrecord, the lock, and/or the success record) from persistent storage, itmay be possible to resume the workflow after the failure. For example,in the workflow of FIG. 42, if MN2 fails after the first atomicoperation and a replacement MNR-2 is designated, MNR2 may read theintent record IR2 and the workflow identifier WFID2 and send a statusquery regarding to MN1, and so on. In a manner similar to that shown inFIGS. 43 and 44, depending on how long it takes to detect the failureand configure the replacement node, and how much progress the renameworkflow had made prior to the failure, in some cases the renameworkflow of FIG. 42 may be abandoned after a metadata node failure. Inembodiments in which metadata is stored using the same underlyingstorage subsystem as is used for data, recovery techniques similar tothose illustrated in FIG. 43 and FIG. 44 may be used to respond tostorage node failures as well. In some embodiments the functionality ofa metadata node and a storage node may be performed at the same host orhardware server, and as a result a failure of that host or server mayaffect both types of nodes.

Scalable Namespace Management

The goals of the distributed storage service may include handling verylarge numbers of files, directories, links, and/or other objects in ascalable manner in various embodiments. For example, for some largecustomers, a given file system may comprise a million or moredirectories, and a given directory may comprise a million or more files.In some embodiments, in order to support high throughputs and/or toensure that response times remain relatively flat at high concurrencyfor various namespace operations such as directory listings, lookups,inserts and deletes as the number of objects in the namespace increasesto such levels, a data structure called a hash-directed acyclic graph(HDAG) may be used for managing namespace entries. The term namespace isused herein to refer to the collection of names of objects (files,directories, hard and soft links, and the like) created within a givenfile system or other data store logical container, and to therelationships (e.g., parent-child relationships) between the objects. Insome embodiments, a respective HDAG may be generated for each directoryof a file system, e.g., by the metadata subsystem of the service. TheHDAG-based namespace management techniques described below may utilizesome of the features of the distributed storage service that have beendescribed earlier, such as the striping of metadata structures atconfigurable granularity across multiple storage extents and the abilityto perform modifications at a plurality of storage devices in a singleatomic operation. For example, in one implementation a respectivelogical block (which may be mapped to one or more pages of a respectiveextent) may be used for each node of a particular HDAG, thus potentiallypartitioning the namespace entries among a plurality of storage servers.

FIG. 45 illustrates an example of a hash-directed acyclic graph (HDAG)that may be used for file store namespace management, according to atleast some embodiments. An HDAG for a directory may include at least twotypes of nodes in the depicted embodiment: entry list (EL) nodes (eachof which comprise a list of directory entries similar to theDFS-DirectoryEntry structures shown in FIG. 39, with pointers torespective DFS-Inodes that contain other attribute values for thecorresponding objects), and node identifier array (NIArray) nodes (eachof which comprise an array of pointers to a set of child nodes). Thetype of a node may be indicated in a header field, such as header field4504A or 4520A. When a directory D1 is created, an HDAG in initial state4590A, comprising a single EL node (such as node 4500A, referred to asthe root node of the HDAG), may be created for the directory. In someimplementations, the DFS-Inode for the directory may itself be used asthe root node of the HDAG. Root node 4500A may comprise sufficient spaceto hold some set of directory attributes 4502A, a header field 4520Rindicating the type of the root node (initially EL), and a root entrylist 4506 for the first few files or subdirectories created within D1. Agiven EL node may store up to some configurable number (e.g., a valuethat may be selected for all the EL entries of a given file store) ofnamespace entries, and a given NIArray node may store up to someconfigurable number of node identifiers (e.g., another value selectedfor all the NIArray entries of a given file store). In at least someembodiments, the maximum permissible size of an HDAG node may bedetermined such that the contents of one HDAG node can be written tostorage in a single atomic operation—e.g., in one implementation, if theHDAG parameters are selected such that an HDAG node never occupies morethan 4 kilobytes, extents that support 4 kilobyte pages may be used forthe HDAGs, and/or a logical block size of 4 kilobytes may be used. Othermappings between HDAGs, logical block sizes, and page sizes may be usedin other implementations.

As more files or subdirectories are added within D1 (as indicated byarrow 4525), the root entry list 4506 may eventually become full, andthe root node 4500A may be split into some number of child nodes using ahash function to distribute its entry list members into the child nodes.The type of the root node may be changed from EL to NIArray, andpointers to the child nodes (e.g., the logical or physical storageaddresses at which the child nodes are stored) may be written torespective elements in an NIArray at the root node. A selected stronghash function may be applied to each of the entry names (e.g., filenames or subdirectory names) to produce a hash value of a desired size,and portions of the bit-sequence representation of the hash value for agiven entry may be used to map the entry to a new child node. Severaltypes of split operations (described in detail below) may be implementedin various embodiments on non-root nodes as they fill up, using asimilar hash-based distribution of entries among newly-created childnodes. In response to lookup requests, the same hash function may alsobe used to search for entries for specified object names, e.g., usingsuccessive subsequences of the bit sequence representation of the hashvalue as indexes to navigate respective levels of the HDAG until a nodewith the targeted entry is found. To obtain a directory listing, all thepointers starting from the root node's NIArray (assuming the root nodehas split) may be followed recursively until the entire HDAG has beentraversed and all its entries have been retrieved. Further detailsregarding various types of HDAG operations are provided below.

The type of an entry list node may change as a result of one or moretypes of HDAG operations under some conditions—e.g., root node 4500A hasbecome an NIArray node after its entries are distributed among childnodes (and as described in further detail below, in some cases anNIArray node may be transformed into an entry list node after adeletion). The NIArray 4510A includes pointers (e.g., storage addresses)of child nodes 4550A, 4550B and 4550C in HDAG state 4590B. The entriesthat were originally in root entry list 4506 may initially bedistributed among respective entry lists at the child nodes (e.g., entrylist 4522A of node 4550A, entry list 4522B of node 4550C, and anotherentry list initially created at node 4550B). Thus, each of the childnodes 4550A, 4550B and 4550C may have started out as an EL node. By thetime state 4590B is reached, however, node 4550B itself has split andbecome an NIArray node, with pointers to its own children nodes 4550Kand 4550L being stored in NIArray 4510B. Node 4550L has also changedstate from EL to NIArray in state 4590B, and its NIArray 4510C includespointers to its children nodes. Node 4550K still remains an EL node,with entry list 4522K representing some of the files/directories createdwithin D1. The headers of each of the nodes (e.g., headers 4520R, 4520A,4520B, etc.) may be modified when and if the type of the node is changedas a result of a node split (or a node join after some types of entrydeletions) in the depicted embodiment. In some implementations, at leastat some points in time, the root node 4500A and/or other HDAG nodes maycomprise some number of bytes that are not in use. In state 4590B, theHDAG may be considered as comprising at least three “levels” including aroot level, HDAG level 1 (comprising nodes 4550A, 4550B and 4550C thatcan be accessed in a single lookup using NIArray pointers of the rootnode), and HDAG level 2 (comprising nodes 4550K and 4550L that can beaccessed in a single lookup using NIArray pointers of level 1 nodes).The term “HDAG level” may be used herein as an indication of the numberof nodes that have been encountered, starting from the root node of theHDAG, to arrive at some particular node. HDAG nodes that have nochildren may be referred to as leaf nodes. At least in some embodiments,it may be the case for two leaf nodes L1 and L2 of an HDAG, duringrespective traversals towards the leaf nodes from the HDAG root,different numbers of nodes may be encountered before reaching L1 thanare encountered before reaching L2. It is noted that in the embodimentillustrated in FIG. 45, the hash values that are used to distribute theentries among the nodes, and thereafter to look up the entries, may notneed to be stored within the HDAG itself.

As noted earlier, one of the goals of the namespace management techniquemay be to enable fast lookups by name. FIG. 46 illustrates a techniquefor navigating an HDAG using successive subsequences of a hash valueobtained for a file name, according to at least some embodiments.(Similar techniques may be used for directories, links or other filestore objects) The name 4602 of the file is used as input to a selectedhash function 4604, e.g., in response to a lookup request with the nameas a parameter. In some embodiments, a string of up to K (e.g., 255)UTF-8 characters may be used as a file name or a directory name. Otherlength restrictions or encodings of file store object names may be usedin other embodiments. In one embodiment, different hash functions may beused for respective file stores—e.g., the hash functions may bespecified as configuration parameters, or may be selected by the storageservice based on expectations of the namespace size for the file store,hints provided by the clients on whose behalf the file store is beingcreated, and so on. In at least one embodiment, various metrics of theeffectiveness of a hash function in use may be tracked over time, suchas the average number of levels of the HDAG for a given number ofnamespace entries, or the degree to which the HDAGs are balanced (e.g.,whether some entries are reached by passing through far fewer levelsthan others), and a different hash function may be selected (at leastfor future use) if the measured effectiveness is not sufficient.

In the depicted embodiment, a hash value 4610 expressible as a sequenceof (at least) N*M bits may be generated, where N and M may beconfigurable parameters. N subsequences of the hash value 4610 (e.g.,S1, S2, . . . SN) of M bits each may be used as indexes intocorresponding levels of the HDAG—e.g., subsequence S1 may be used toselect the NIArray pointer (of the root node) to be used to navigatelevel 1, subsequence S2 may be used to select the NIArray pointer to beused to navigate level 2 starting from the level 1 node, and so on. Notall the bits in a given subsequence need be used for a given search ornavigation level—e.g., only the q high-order bits (where q<M) may beused in some cases. In some embodiments, some bits 4666 of the hashvalue may not be used for any level.

When a new entry is to be added to a file store, e.g., in response to anopen file command or create directory command, the hash value for thename of the new entry may be obtained, and the HDAG may be traversedusing the subsequence-based navigation technique described above until acandidate EL node to which the name is mapped is found. (In somescenarios, it may be the case that the namespace has run out of spacefor entries—such special cases are discussed below). If the candidatenode has no more free space in its entry list, or of its free spacewould fall below a threshold level if the new entry were added, thecandidate node may be split. At least some of the entries of node thatis split may be distributed among one or more new nodes added to theHDAG, e.g., using selected subsequences of the hash values of theentries as described below. At least two different types of HDAG nodesplit operations may be performed in some embodiments.

FIG. 47 illustrates an example of the first of two types of HDAG nodesplits that may result from an attempt to insert an entry into anamespace, according to at least some embodiments. In this first type ofsplit, the type of an HDAG node may be changed from entry list (EL) toNIArray as described in detail below. The namespace entry insertion maybe one of several steps taken in response to a client request to createa namespace object such as a file in some embodiments—e.g., the othersteps may include allocating space for a DFS-Inode object associatedwith the file, setting the initial attributes of the file and/or settinga pointer from the namespace entry to the DFS-Inode and from the Inodeto one or more physical pages to be used for storing file contents. Theorder in which these steps are taken may differ in differentembodiments.

A request to insert an entry 4701 with name (e.g., file name) “Lima”into a namespace is received in the embodiment shown in FIG. 47, and acandidate EL node 4750A is found after navigating within the HDAGcreated for the directory into which the insertion of the object withname “Lima” is being attempted. Initial portions of the identifiers ofthe HDAG nodes (which may also correspond to their storage addresses,and thus may be used as parameters to read or write operations directedto the storage subsystem) are shown as hexadecimal strings in FIG.47—e.g., node 4750 has an ID “0x432d12 . . . ”. The first type of nodesplit, illustrated in FIG. 47, may be attempted under the followingconditions in the depicted embodiment: either (a) the candidate node4750A is the root node or (b) only one NIArray pointer entry in theparent node of node 4750A (not shown in FIG. 47) points to node 4750A.If either of these conditions is met, space may be allocated (e.g., atrespective metadata extents) for two new HDAG nodes 4750B and 4750C inthe depicted embodiment. (It is noted that two child nodes areillustrated in FIG. 47 for ease of presentation; in other embodiments,more than two new child nodes may be created during a split.) Each ofthe entries that were previously in node 4750A (e.g., “Alpha”. “Bravo”,“Charlie”, etc.), and the new entry “Lima”, may be mapped to one of thenew nodes 4750B or 4750C based on their respective hash values, asindicated by the arrows labeled “1”. In one implementation, for example,if the candidate node were in the Kth level of the HDAG, the (K+1)thsubsequences of the hash values for the entries may be sorted based ontheir most significant bit, and the entries whose hash values have “1”as their most significant bit may be mapped to node 4750B, while theentries whose hash values have “0” as their most significant bit may bemapped to node 4750C. In embodiments in which more than two child nodesare created during a split, more bits may be used for the mapping of theentries—e.g., if four child nodes are created, the two highest-orderbits of the hash subsequence values may be used, and so on. In thedepicted embodiment, depending for example on the object names and thehash function, it may not always be the case that the entries of thenode being split (4750A in the depicted example) are distributeduniformly between the child nodes, and at least in some embodiments noattempt may be made to “balance” the HDAG by trying to achieve suchuniformity. Instead, the strength and quality of the hash function maybe relied upon in such embodiments to achieve a reasonably balanceddistribution of entries among the nodes. After the distribution of theentries among the child nodes in the depicted example, child node 4750Bhas free space 4710A that may be used for subsequent insertions, whilechild node 4750C has free space 4710B that may be sued for subsequentinsertions.

Node 4750A, which was an EL node prior to the split, may be convertedinto an NIArray node, as indicated by the arrow labeled “2” in FIG. 47.Half of its NIArray entries may be set to point to node 4750B (e.g., bystoring 4750B's ID 0x786aa2 . . . ) and the other half may be set topoint to node 4750C (e.g. by storing 4750C's ID 0xc32176 . . . ). In animplementation in which the most significant bit was used to split theentries, the lower half of the NIArray entries (e.g., entries withindexes 0 to (NIArraySize/2)−1) may be set to point to the node 4750C(entries whose hash values began with “0”), and the upper half of theNIArray entries (e.g., entries with indexes (NIArraySize/2) to(NIArraySize−1)) may be set to point to the other child node 4750C. Inembodiments in which n children nodes are created as a result of thesplit, 1/n of the NIArray entries may be set to point to each of thechildren. The changes to the three nodes 4750A, 4750B and 4750C may besaved to persistent storage at the storage subsystem. In someembodiments, changes to all three nodes may be performed in a singleatomic operation, e.g., using the distributed transaction techniquedescribed earlier. In other embodiments, the conditional writesdescribed earlier may be used to make the changes for at least one ofthe three nodes persistent separately from the other nodes.

If the conditions outlined above for performing the first type of splitoperation are not met (e.g., if the parent node of the candidate nodehas more than one NIArray pointer to the candidate node), a second typeof split operation may be performed. FIG. 48 illustrates an example ofthe second of two types of HDAG node splits that may result from anattempt to insert an entry into a namespace, according to at least someembodiments. In the depicted example, node 4750C has been identified asthe candidate node for a new entry “Queen” 4801, and node 4750C has nofree space left in its entry list. The parent node, 4750A, includesnumerous pointers to node 4750C (e.g., the NIArray entries with the IDvalue 0xc32176 . . . ) at the time the insert of “Queen” is attempted.As indicated by the multiple elements with the same value “0x786aa2 . .. ”, and the multiple elements with the value “0x32176 . . . ”, in thedepicted embodiment, the NIArray elements each point to the block atwhich the node's content is stored, not to individual EL entries withinthe node. In other embodiments, entry-level pointers may be used insteadof or in addition to block-level pointers. In the scenario depicted inFIG. 48, only one new node (node 4850A with ID 0x223123 . . . ) iscreated instead of two nodes as was illustrated in FIG. 47. Hash valuesfor the entries of node 4750C may be computed in a manner similar tothat used for 4750A entries in FIG. 47. The hash values may be sortedbased on the most significant bit. Those of the entries in 4750C at thetime of the split that have a “1” as the most significant bit may bemapped to the new node 4850A, while the remaining (the ones with “0” asthe most significant bit) may be kept within node 4750C, as indicated bythe arrow labeled 1.

The parent node's NIArray entries may be modified to add pointers to thenewly-added node 4850A in the depicted embodiment, as indicated by arrow2. Of the 4750A NIArray entries that were previously pointing to 4750C,one half (e.g., the upper half of the array index range) may be set topoint to the new node 4850A, while the other half may continue to pointto 4750C. Thus, after the split, among the NIArray entries of node4750A, half may contain the ID of 4750B (which was not affected in thesplit), one quarter may point to 4750C, and one quarter may point to4850A. As in the case of the first type of node split discussed above,in some embodiments, the entries of the candidate node 4750C whose EL isfull may be redistributed among more than two nodes (including thecandidate node itself)—e.g., a total of 4 nodes may be used using 2 bitsof the entry hash values for the distribution. Under some circumstances,a split of a given node may have to be propagated upwards towards theroot of the HDAG—e.g., a node N1 may have to be split due to an insert,as a result N1's parent may also have to be split, and so on. Theprocedure of traversing the HDAG to reach a candidate node may have tobe repeated in such cases, starting from the root of the HDAG.

The split operations illustrated in FIGS. 47 and 48 assume that a newlevel (e.g., new child pointers) may be added to the HDAG at the timewhen the split is attempted. However, in at least some embodiments,based for example on the hash value size and the number of bits used fornavigating each level of the HDAG, at some point the maximum number oflevels allowed by the hash function may be reached, and no more levelsmay be added. In such a scenario, instead of performing the hash-basedsplits illustrated in FIGS. 47 and 48, a chain or linked list for newentries that cannot be accommodated by the hash-based split may becreated (e.g., using a third type of HDAG node). For example, in FIG.48, if node 4850 becomes full and the limit on the number of levels hasbeen reached when an attempt to insert a node “Tom” is made, a new nodeof type “chain” may be created to store “Tom”'s entry, and a pointer tothe chain node may be inserted at a selected location in the candidatenode. The chain node may itself be modified to point to other chainnodes if needed. In order to locate any given entry that has beenincluded in a chain node, a sequential scan of the chain may be usedinstead of a hash-based lookup as is used at other types of nodes. Inthis way, large numbers of entries may be accommodated even if the HDAGbecomes “unbalanced”, although of course some of the speed advantages ofhash-based traversal may be lost, as the chained entries may have to betraversed sequentially for a lookup. In various embodiments, theselection of a reasonably long hash value and a strong hash function mayreduce the probability of having to use chain nodes to below anacceptable threshold.

When a namespace entry E is to be deleted (e.g., when the correspondingfile or directory is deleted at a client's request), the EL node fromwhich the entry is to be deleted may be found using the hash-basedtraversal technique outlined above, in which respective subsequences ofthe hash value for the name of the object are used as indexes atsuccessive levels of the HDAG. The EL node from which the entry is to beremoved may be referred to as the deletion target node. If the deletiontarget contains more than one entry, E's entry may simply be deleted ormarked as free, and no additional operations may be required. However,if there were no other namespace entries at the deletion target (i.e.,if removing E's entry would result in an empty entry list), then thedeletion target node itself may have to be deleted. FIG. 49 illustratesan example of the first of two types of HDAG node deletion operations,according to at least some embodiments. In the depicted example, arequest to delete “Juliet” from a namespace represented by an HDAG isreceived. A hash value for “Juliet” is computed, and successivesubsequences of the hash value are used to navigate from the root of theHDAG towards node 4950. Node 4950 is an EL node with a single entry (theentry for “Juliet” that is to be deleted) remaining. The Juliet entrymay be deleted (as indicated by the “X” symbol and the accompanyinglabel “1”.) Because removing Juliet's entry results in an empty entrylist at node 4950, node 4950 may itself have to be deleted. Theconsequences of deleting node 4950 on its parent node 4948 may differdepending on the state of node 4948's NIArray list.

In the depicted embodiment, the deletion target node's parent node mayin general have one or more NIArray elements that point to the deletiontarget node (which may be termed “deletion target pointers”), and zeroor more NIArray elements that point to nodes other than the deletiontarget node. Those NIArray elements that point to nodes other than thedeletion target node, and are next to the deletion target pointerswithin the NIArray (e.g., at the immediately adjacent lower indexeswithin the array) may be termed “neighbors” of the deletion targetpointers. If at least one neighbor exists in 4948's NIArray list whenthe last entry of the deletion target node is deleted, the neighborpointer values may simply be copied into the deletion target pointers inthe depicted embodiment. In the scenario depicted in FIG. 49, forexample, there are two deletion target pointers, 4901 and 4902, inparent node 4948 that point to the deletion target node 4950 (asindicated by the fact that 4950's ID 0xc44321 . . . is stored in 4901and 4902). Also, parent node 4948's NIArray comprises a neighbor element4903, which stores a node ID 0x32176 . . . . Thus, as indicated by thearrow labeled 2, when a deletion of the Juliet entry results in an emptyentry list at deletion target node 4950, and parent node 4948 comprisesat least one neighbor in its NIArray, the contents of that neighbor arecopied into the NIArray entries that were previously pointing to thedeletion target node 4950. In addition, in the depicted embodiment, thedeletion target node 4950 may be freed, e.g., by sending a request torelease its storage space to the storage subsystem. The replacement ofthe contents of the deletion target pointer array elements by thecontents of the neighbor pointer is indicated by arrow 4904. It is notedthat in different embodiments, different techniques may be used todesignate neighbors of the deletion target pointers—in some embodimentsthe NIArray entry that has the next higher index within the NIArray maybe selected as the neighbor, for example.

If there were no neighbors in the NIArray entry of the parent node ofthe deletion target node, the parent node may be reorganized in adifferent way in some embodiments. FIG. 50 illustrates an example of thesecond of two types of HDAG node deletion operations, according to atleast some embodiments. As shown, the deletion target node 4950comprises a single entry in its entry list. That sole remaining entry(“Juliet”) is deleted, as indicated by the “X” symbol and theaccompanying label “1”. In the depicted example scenario, the NIArray ofparent node 4948 does not contain any neighbor elements (i.e., NIArrayelements that do not point to the deletion target node). The approachillustrated in FIG. 49 may thus not be feasible, as there are noneighbor pointer values available. Accordingly, a different approach maybe taken, as illustrated by the arrow labeled “2”: the type of theparent node 4948 may be changed to EL (entry list) instead of NIArray,and an empty entry list may be initialized for node 4948. Thenewly-initialized EL node may be re-used, e.g., when a new node is to beadded to the HDAG as a result of the types of split operations describedearlier. The deletion target node 4950 may be freed, in a manner similarto that discussed above with respect to FIG. 49. In various embodiments,the modifications made at a given level of an HDAG may in some casesrequire changes at other levels—e.g., in one embodiment, when the typeof node 4848 is changed as described above, 4848's parent node's NIArrayentries may have to be modified, and the effects of the changes maypropagate upwards towards the root of the HDAG. As mentioned earlier, invarious embodiments the conditional write technique and/or thedistributed transaction technique described earlier may be used tocombine a desired number of the HDAG changes resulting from a giveninsert or delete into an atomic operation.

FIG. 51 is a flow diagram illustrating aspects of operations that may beperformed in response to an insertion of an entry into a namespace thatresults in a first type of HDAG node split, according to at least someembodiments. A simple example of such a split operation is provided inFIG. 47. As shown in element 5101, a request to add an entry E to anamespace of a distributed multi-tenant storage service is received. Therequest may be generated, for example, in response to a command tocreate a file “Fname”, or open a file “Fname”, issued by a client of afile system implemented at the service. In one embodiment, the requestmay be generated at a command interpreter component at a particularmetadata subsystem node, and may be received at a namespace managercomponent at another metadata subsystem node (or at the same metadatasubsystem node). A hash function may have been selected for namespacemanagement for the targeted file system (e.g., based on the strength ofthe hash function, the expected size and/or performance requirements ofthe file store, and/or on other factors). The hash function may be usedto generate a hash value Hvalue corresponding to “Fname”, where Hvaluecan be expressed as N subsequences of M bits each (element 5104). In oneimplementation, for example, Hvalue may comprise 8 subsequences of 8bits each, thus consuming at least 64 bits.

An HDAG comprising at least two types of nodes (node identifier array(NIArray) nodes and entry list (EL) nodes as described earlier) may havebeen set up for the namespace, e.g., for the directory into which thenew file Fname is being added. An entry list node may be able toaccommodate up to Max-EL entries in the depicted embodiment, whereMax-EL may depend on such factors as the maximum lengths of the objectnames supported, the length of the DFS-Inode addresses or identifiersstored in the entry list, the number of bytes being used for an HDAGnode, and so on. Similarly, an NIArray may be able to accommodate up toMax-NIDs elements in the depicted embodiment, with Max-NIDs beingdependent upon the size of the node IDs and the size of the HDAG nodes.In at least one embodiment, a threshold population of entriesEL-threshold may be designated, such that if the number of entriesexceeds EL-threshold as a result of an insertion, a node split is to beinitiated. In some implementations, the default value for EL-thresholdmay be set to Max-EL, e.g., splits may only be implemented when the ELbecomes full. Similarly, a threshold may be defined for NIArray nodes inat least one embodiment, e.g., when the number of elements in theNIArray at a node exceeds NID-threshold, the NIArray node may be split.NID-threshold may be set to Max-EL by default in some embodiments.Either EL-threshold, NI-threshold, or both El-threshold and NI-thresholdmay be implemented as configurable parameters in some implementations.

Starting from the root of the HDAG (the zeroth level), one or more HDAGlevels may be navigated or traversed to identify a candidate node CNinto which E should be added, using successive M-bit subsequences ofHvalue to identify the specific node or nodes to be examined at eachlevel (element 5107). In at least some embodiments, each of the nodes ofthe HDAG may correspond to a different logical block, and theprobability that a different extent at a different storage subsystemnode is being used for it than for the other HDAG nodes may be high. Ifno candidate node is found (which may in some cases happen if themetadata subsystem has run out of space for the HDAG), as determined inelement 5110), an error may be returned (e.g., “maximum number of filesallowed in a directory has been exceeded”) (element 5113). If acandidate node CN is found (as also determined in element 5110), and itsentry list has enough space to accommodate the new entry E (e.g., theaddition of E would not cause the EL length to exceed EL-threshold) (asdetected in element 5116), the new entry E may be written to one of thecurrently unused entries in the list (element 5119). The modification toCN may be saved to persistent storage in the depicted embodiment, e.g.,at one or more metadata extent replicas. In at least some embodiments, aDFS-Inode structure may be allocated for the object with name Fname, anda pointer to that DFS-Inode structure may be included within E. Inresponse to subsequent lookup requests for “Fname”, hash-basednavigation similar to that illustrated in elements 5104 and 5107 may beused (i.e., respective subsequences of the hash value obtained for“Fname” may be used for respective levels of HDAG navigation until theentry for “Fname” is found).

If CN does not have enough space for E (e.g., if the EL-threshold hasbeen reached, or would be reached by the insertion of E) (as alsodetected in element 5116), the number of pointer's in CN's parentNIArray list that point to CN may be determined. If the parent node hasonly one pointer to CN (or happens to be the root node of the HDAG) (asdetected in element 5122), a first type of node split operation (similarto that illustrated in FIG. 47) may be initiated. Respective hash valuesmay be obtained for the object names in each of the entries in CN's list(element 5125), in addition to the Hvalue already obtained for the newentry E. The hash values may be used to distribute the entry listmembers and E into P groups in the depicted embodiment (element 5128),e.g., using the log 2P most significant bits of the hash values as thesorting/distribution criterion. In one example implementation, P may beset to 2, so only the single most significant bit may be used. Each ofthe P groups may be stored as an entry list of a respective new node tobe added to the HDAG (element 5131). A new NIArray may be created, withapproximately 1/Pth of the array elements pointing to (e.g., containingthe storage addresses or identifiers of) each of the P new nodes. CN'sheader may be modified to indicate that it is an NIArray node ratherthan an EL node, and the new NIArray may be written into CN (element5134). The contents of the P new nodes of the HDAG and the modified CNmay be saved to persistent storage, e.g., at one or more storagesubsystem nodes. In some embodiments, the distributed transactiontechnique described above may be used to combine some subset or all ofthe changes to the HDAG into a single atomic operation. In otherembodiments, conditional writes of the type described earlier may beused for at least some of the HDAG nodes.

If the number of NIArray elements that were pointing to CN from CN'sparent node exceeded one (as also detected in element 5122), a secondtype of split operation may be conducted on CN (as indicated by the “Goto 5201” element of FIG. 51). FIG. 52 is a flow diagram illustratingaspects of operations that may be performed in response to an insertionof an entry into a namespace that results in such a second type of HDAGnode split, according to at least some embodiments. This type of splitmay be designated as a type-2 split herein, and the type of splitillustrated in FIG. 51 may be referred to as a type-1 split. In thetype-2 split, some of the members of CN's entry list may be moved into Qnew HDAG EL nodes (where Q is no less than one), while some may remainin CN, and the parent node's NIArray pointers may be changedaccordingly. In the depicted embodiment, a sub-list of CN's entry listmay be selected for redistribution among Q new HDAG nodes NN1, NN2, . .. NNQ and in CN itself. In one implementation, Q may be set to 1 andapproximately (or exactly) half of the entry list may be considered forredistribution, while in another implementation, three-fourths may beconsidered. A respective hash value may be determined for each member ofthe sub-list (element 5204). The hash values may be used to arrange thesub-list members into Q+1 groups (element 5207), e.g., using some numberof most significant bits of the hash values as the distributioncriterion.

Q of the groups may be placed in respective new HDAG EL nodes, while theremaining group may be retained within CN. Some of the NIArray entriesin CN's parent node that were pointing to CN may be set to point to thenew nodes NN1, NNQ (element 5210). In the depicted embodiment, the HDAGnodes that were modified or created as a result of the split (e.g., theQ new nodes, CN, and CN's parent node) may be written to persistentstorage in a single atomic operation (element 5213). The distributedtransaction technique described above may be used in some embodiments.In other embodiments, a single atomic operation may not be used; forexample, the conditional write technique may be used for at least someof the HDAG nodes.

It is noted that the technique whereby entry list members arere-distributed in type-2 splits may differ in some embodiments from thatillustrated in FIG. 52. For example, in some embodiments, the sub-listmembers may be selected in such a way that they may be distributedentirely among the Q new nodes. In some embodiments, the size of thesub-list may be chosen at random—e.g., not all the type-2 splits thatare implemented at a given HDAG or at a given file store may result inthe same number of new nodes. In some embodiments, an element ofrandomness may also be introduced into type-1 splits—e.g., theEL-threshold used may be varied at random within a range, or the numberof new nodes P may be selected at random from a range.

FIG. 53 is a flow diagram illustrating aspects of operations that may beperformed in response to a deletion of an entry from a namespace,according to at least some embodiments. As shown in element 5301, arequest to remove an entry E for a file store object with a name Fnamefrom a namespace of a distributed storage service may be received. Sucha request may be generated as a result of a client request to remove afile or directory, for example. Using a selected hash function, a hashvalue Hvalue whose bit sequence can be divided into N subsequences of Mbits each may be obtained (element 5304).

An HDAG generated for the namespace may be navigated or traversed,starting from its root node, to identify a deletion target node N1 whichcontains E (element 5307). At each level of the HDAG, a successivesubsequence of the N subsequences may be used to identify the nodes tobe read or examined. If N1's entry list includes at least one more entry(as detected in element 5310), E's slot within the entry list may simplybe marked as unused or free (element 5313) and the deletion operationmay be completed. In some implementations, e.g., to make it quicker tofind non-empty entries, the freed entry may be moved to one end of thelist. Thus, for example, if an entry list of length N contains twonon-empty entries, in one such implementation, those two non-emptyentries would be found at offset 0 and offset 1 within the list, whilethe entries with offsets 2, 3, . . . , N−1 would be empty. In someembodiments, the change to N1 may be made persistent synchronously,while in other embodiments N1 may be written to persistent storage atone or more extents asynchronously with respect to the delete requestfor E.

If E was the last entry in N1's entry list (as also detected in element5310), the NIArray of N1's parent node PN may be examined. PN's NIArraymay comprise one or more elements NP1, NP2, . . . , pointing to (e.g.,storing the address or identifier of) N1. If the NIArray of PN alsoincludes at least one “neighbor” element NX that points to some othernode than N1 (as determined in element 5316), the contents of NX may becopied to NP1, NP2, . . . so that PN no longer contains a pointer to N1(element 5319). In at least some embodiments, the array elements NP1,NP2, . . . may also or instead be marked as invalid.

If PN's NIArray contains no such neighbor elements that point to nodesother than N1 (as also detected in element 5316), PN may be modified ina different way in the depicted embodiment. As shown in element 5322,PN's type may be changed from NIArray to EL, e.g., by modifying itsheader. In addition, a new entry list may be initialized for PN—e.g., atleast some of the bytes that were being used for the NIArray may beoverwritten. In the depicted embodiment, regardless of whether aneighbor element was found or not in the parent node PN, the deletiontarget node may be marked as free or unused (element 5325). Contents ofeach of the node affected by the deletion, e.g., PN and N1, may be savedto persistent storage at one or more extents of the storage subsystem.In some embodiments a distributed transaction of the type describedearlier may be used to make at least the changes shown in elements 5322and 5325 part of a single atomic operation. In another embodiment, themodifications shown in element 5319 may also be combined with those ofelements 5322 and 5325 in a single atomic operation or distributedtransaction. Conditional writes may be used for each of the changes inat least one embodiment.

In various embodiments, configurable parameters (e.g., defined either atthe file system level, or for the file storage service as a whole) maybe used to determine various aspects of the hash-based namespacemanagement technique. Such configurable parameters may be specified forany combination of: (a) the specific hash function(s) or hash functionfamily to be used, (b) the required lengths of the bit sequence outputby the hash function, (c) the lengths of various subsequences of thehash value output to be used for traversing respective levels of theDAG, (d) the fan-out of the splits of each type (e.g., the number oflists to which the entries of the split node are to be assigned in eachsplit type), (e) the number (or fraction) of NIArray elements in whicheach new node's identifier is to be stored after a split, (f) thethreshold population levels for each type of split, or (g) the maximumpermissible number of levels of the DAG or the total size of the DAG. Insome embodiments, additional constraints (e.g., extent placementconstraints) may also be specified via parameters—e.g., a constraintthat all the HDAG nodes of the first N levels be stored at the sameextent may be specified, or a constraint that no two HDAG nodes shouldbe stored at the same extent may be specified. In some embodiments, oneor more of these parameters may be modified based on collectedperformance results. E.g., if namespace-related performance isunsatisfactory with a given set of parameters for a particular filesystem, the storage service may adjust the parameters—either for thesame file system (which may involve new HDAGs to be created either onthe fly or during a reconfiguration downtime period) or for file systemscreated subsequently.

Client Session Metadata Management

In at least some embodiments, the distributed storage service maysupport one or more stateful or session-oriented file system protocolssuch as NFS. In some such protocols, a client component of the service(e.g., a daemon running at a client-side execution platform) maytypically create a session via one or more communications with a servercomponent (e.g., another daemon running at a server-side executionplatform), where the session has an associated expiration time duringwhich the service is able to expedite responses to certain kinds ofclient requests, and where the session may be extended or renewed undersome conditions. During a session, the client may, for example, obtain alock on an object such as a file, and the lock may remain in effectuntil either the session ends or the client releases the lock.Subsequent accesses of the object from the client during the session maynot require additional locking. According to some file system protocols,such a time-bound grant of control of the state of a file (or anothertype of file store object) to a client from the server may be referredto as a “lease”. A single lease may be associated with locks on aplurality of file store objects, and may be renewed either explicitly orimplicitly by the client. In at least some embodiments, asession-oriented protocol may require that session state information(e.g., a list of locked files or directories associated with a client'slease, the expiration time of the lease, and so on) be maintained by the“file server”. In a distributed file storage service, theprotocol-mandated responsibilities of the file server may be distributedamong the various subsystems described above—e.g., the access subsystem,the metadata subsystem, and/or the storage subsystem. Various factorssuch as scalable response time and throughput goals, metadata durabilityrequirements, and so on, may be taken into consideration when decidingthe specific portions of the protocol-mandated session-relatedfunctionality that should be implemented at different subsystems indifferent embodiments.

FIG. 54 illustrates two dimensions of metadata that may be maintainedfor session-oriented file system protocols at a distributed storageservice, according to at least some embodiments. Information about allthe objects that have been opened and/or locked during a given clientsession may have to be accessed efficiently by the storage service forcertain types of operations (e.g., for lease expirations, which mayrequire that all the locks of a session be released). This firstdimension of metadata information is represented by a row in theconceptual metadata table 5401 shown, such as the contents of metadataset 5401 that may be accessed for lease-related operations on clientsession CS1. Metadata set 5401 may, for example, comprise lock stateindicators (LSIs) (such as NFS “StateIDs”) whose use is discussed infurther detail below, for a plurality of files, directories, links andthe like. In the example shown, for client session CS1 a write lockstate indicator W-lock is shown for directory D1, and R-locks (read lockindicators) are shown for files F1 and FP. It is noted that at least insome implementations, locking may be implemented at the file level butnot at the directory level.

The second dimension is the set of session-related information that hasto be maintained in accordance with the file system protocol on anygiven object, such as metadata set 5420 on file F1. This secondcollection of metadata (which may also include lock state indicatorssuch as the R-lock of client session CS1) may have to be accessedefficiently when, for example, a new request to lock the object isreceived, or when a request to view the state or attributes of theobject is received. In a file store that may store millions of objects(many of which are at least potentially distributed across multipleextents) and may have tens of thousands of concurrent client sessionswith many different types of locking modes and/or leasing modessupported, it may not be practical or efficient to store all of thesession-related information of the type illustrated in FIG. 54 in asingle centralized location. FIG. 54 thus provides a conceptual view ofat least two kinds of session-related metadata that may have to beaccessed efficiently in various embodiments, and is not intended toimply any particular implementation approach.

It is noted that in addition to the session-oriented metadata 5401required by a particular file system protocol, other internal metadata(such as namespace management metadata including HDAGs as describedabove, logical-block-to-physical-page mappings as described earlier,etc.) may also be maintained. The different types of metadata may bemanaged by independent subcomponents of the metadata subsystem in atleast some embodiments—e.g., the management of striping orlogical-block-to-physical-page mappings may be implemented orthogonallywith respect to the management of client session information of the typeillustrated in FIG. 54. Furthermore, the distributed storage servicemay, at least in on embodiment, support a plurality of stateful orsession-oriented file system protocols, each of which might definerespective session metadata object types and semantics. For example, NFSmay specify its set of metadata objects and relationships, SMB mayspecify a different set, and so on. In such scenarios, separate sets ofsession-oriented metadata 5401 may be maintained for file systemsassociated with each of the different protocols.

In at least some embodiments, a client (such as an NFS clientimplemented using one or more processes at a compute instance of aprovider network) may request an establishment of a client session bytransmitting a message to the distributed storage service, formatted inaccordance with the file system protocol. FIG. 55 illustrates an exampleof client session metadata-related interactions between subcomponents ofa distributed storage service, according to at least some embodiments.File system client 5501 may send a session request 5550 to an accesssubsystem node 5512, e.g., an access subsystem node whose IP address hasbeen exposed or advertised as an endpoint for the file system being usedby the client. In some implementations in which the file system protocolbeing used is NFS, for example, the session request may comprise a“SetClientID” request, and may include an identification of the client(generated by the client) and a unique, non-repeating object called a“verifier” (also generated by the client). The verifier may be used insome such implementations by the service to determine whether a clienthas rebooted since the session was originally instantiated; thus, thesubmission of a second SetClientID request with a different verifier mayallow the service to expire the client's earlier session/lease. Inresponse to the session request, the file system protocol in use mayrequire that (unless error conditions are encountered) a sessionidentifier 5563 (e.g., an NFS “ClientID” object) ultimately be providedto the requester by the service.

In at least some embodiments, the metadata subsystem of the distributedstorage service may be responsible for managing the client session stateinformation. For example, the metadata subsystem may control the mannerin which client session state information is mapped to logical blocks aswell as the mapping of those logical blocks to extents. The extentsthemselves may be stored at storage subsystem nodes in some embodiments,and at the metadata subsystem nodes in other embodiments as describedearlier. While the access subsystem nodes may cache session-relatedmetadata temporarily in some embodiments, the metadata subsystem may bedesignated as the authoritative source of client session informationwithin the distributed storage service.

In the depicted embodiment, upon receiving the client session request,the access subsystem node 5512 may transmit a session initializationrequest 5553 to a selected metadata node 5522, requesting a sessionidentifier to be generated by the metadata subsystem. The parametersprovided by the client (e.g., the client's identifier and/or verifier)may be passed along to the metadata node by the access node in at leastsome embodiments. The metadata node 5522 may generate a new logicalblock LB1 to store at least a portion of the client's session metadata.LB1 may include, for example, a session identifier 5563 generated forthe client session by the metadata node, a lease timeout setting 5544for the session, and a “responsible access node” (RAN) field 5546 in thedepicted embodiment. The RAN field may identify the particular accessnode 5512 through which the client's requests during the ensuing sessionare expected to be received at the back-end subsystems (e.g., themetadata subsystem or the storage subsystem). The metadata node 5522stores contents of the logical block of the session metadata at one ormore pages of a selected extent 5580 in the depicted embodiment, asindicated by arrow 5557. In some implementations, the metadata node 5522may submit a request to the storage subsystem to store the logical blockcontents, while in other embodiments, the metadata node 5522 may writethe contents to an extent that is managed by the metadata subsystemitself.

According to at least some embodiments, the session identifier (e.g.,NFS ClientID) selected or generated for the client may be based at leastin part on the storage address of the logical block—e.g., the sessionidentifier may be used later as a parameter in a read operation toquickly look up the client session metadata. For example, in oneimplementation, each logical block may be assigned a 128-bit logicalstorage address, and the 128-bit logical address used for LB 1 may beprovided as the session identifier 5563 for the client, or may beincluded or encoded within the session identifier 5563. In anotherembodiment, the session identifier may be based at least in part on thephysical storage address of at least one of the physical block(s) beingused to store the session metadata elements. The metadata node 5522 maytransmit a response 5560 to the session initialization request 5553. Theresponse 5560 may include the session identifier 5563, which may becached at the access node 5512 at cache 5578 and provided to therequesting client 5502 in the depicted embodiment. In some embodiments,the file system's session establishment protocol may require one or moreaddition interactions, e.g., a confirmation request message comprisingthe session identifier may be sent to the storage service by the client5502 and the client may then receive a response confirming the validityof the session identifier. Subsequent requests from the client, such asfile opens, closes, lock requests and the like may be required toinclude the session identifier 5563 in at least some embodiments. Onreceiving such later requests, the access node 5512 may validate theclient's session identifier using cache 5578. If the session identifieris missing from the cache, the access node may submit a query to themetadata subsystem regarding the session, and may only proceed with therequested operation if the session is still open (or if a new session isinstantiated by the metadata subsystem in response to the query).

As indicated earlier, in some embodiments a file system protocol such asNFS may implement a leasing technique for efficiently managingconcurrent accesses to file system objects. In some such embodiments, alease associated with a client session may represent a time-bound grantof control of the state of one or more files, directories, links orother client-accessible objects of a file system to the client. In atleast one embodiment, another metadata object, referred to herein as alock state indicator, may be used to represent the locking state of aparticular file system object by the storage service. For example, in atleast some implementations of the NFS protocol, a lock state indicatormay be termed a “StateID”. A lock state indicator for an object such asa file F1 may be defined in at least some embodiments in the context ofto a given client session CS. Thus, for example, when a client Cl1 locksa file F1 as part of a client session CS1, a lock state indicator LSI1for F1 that is specific to CS1 may be created; and later, when adifferent client Cl2 locks file F1 as part of a client session CS2, adifferent lock state indicator LSI1 may be generated by the storageservice. In at least some embodiment, an LSI may incorporate, or includea pointer to, the session identifier of the corresponding clientsession—e.g., in one implementation, an NFS-compliant StateID mayinclude a pointer to (or the actual value of) the correspondingClientID. Each open client session may have an associated lease timeoutperiod in some embodiments, at the end of which the locks associatedwith all of the session's LSIs may be freed. In some embodiments, openstate indicators (similar to LSIs) may be used to indicate that aparticular file store object is currently open for access by a client.An indication of the open state and the locked state of a file storeobject may be represented using a single metadata structure (e.g., anopen/lock state indicator) in some implementations.

According to the semantics of at least some file system protocolsimplementing leases, one or more mechanisms for lease renewals may besupported. For example, a set of operation types may be defined, suchthat a request for an operation of that set of operation types by aclient during an open session may automatically result in the renewal ofthe lease for some specified lease renewal term. If a client issues arequest to read a file F1 in such an embodiment, for example, during asession CS1 for which the lease was set to expire at time T1, the leasemay be extended to a later time T2. In some embodiments, APIs forexplicitly renewing leases may also or instead be supported. If none ofthe types of requests that result in automatic (or explicit) leaserenewal are received for a specified period, the lease may expire. Insome embodiments, upon lease expiration, the corresponding locks(indicated by LSIs) may be released by the storage service, file systemobjects that were opened during the session and had not been closedbefore the lease expiration point may be closed, and at least in someembodiments the session metadata may be deleted from the metadatasubsystem's persistent repository and/or from the access subsystem'scaches.

FIG. 56 illustrates alternative approaches to client session leaserenewal at a distributed storage service, according to at least someembodiments. In the depicted embodiment, an auto-renew operation list5678 may be specified by a file system protocol being used by theclient. The auto-renew operation list 5678 may indicate operation typesthat when requested during a currently open session, result in theautomatic renewal of the lease(s) associated with the session. Forexample, in some NFS implementations, the auto-renew operation list mayinclude (among others), read, write, open, lock, unlock, andset-attributes operations. In some implementations, a renew operationfor explicit renewal of a lease may also be included in the operationlist 5678.

In the depicted embodiment, an access subsystem node 5512 may receive afile store operation request 5650. If the operation request is of a typeindicated in the auto-renew operation list (or is an explicit request torenew the client's lease), the access node 5612 may have two options inthe depicted embodiment. The access node may either submit an immediateor un-batched lease renewal request 5653 to the metadata node 5522, ormay defer the lease renewal for up to some configurable time period andsubmit a batched lease renewal request 5654 to the metadata node 5522.The batched lease renewal request may, for example, comprise sessionidentifiers for a plurality of client sessions for which auto-renewaloperation requests or explicit renewal requests were received during atime window. The batching of lease renewal requests may help to reducethe renewal-related overhead (e.g., communication overhead, processingoverhead, or both) at the metadata node 5522 and/or the access node 5512in at least some embodiments.

In some embodiments, a configurable immediate renewal threshold 5688 maybe used by the access node to determine whether a given lease renewalshould be transmitted immediately in response to the client's operationrequest 5650, or whether the deferred batch approach should be used forthe client's lease renewal. If the immediate renewal threshold is set toX seconds, for example, and the client's lease is set to expire within Xseconds of the time that operation request 5650 is received by theaccess node, an un-batched or immediate lease renewal request 5653 maybe generated in the depicted embodiment. Otherwise, if more than Xseconds remain before the lease is set to expire, a representation ofthe client's renewal request may be stored in batched renewals buffer5679, and some number of renewals may be sent later in a batched leaserenewal request 5654 to the metadata node 5522. The access node may havecached the lease expiration times for various client sessions for whichthe access node is responsible within session metadata cache 5578 in thedepicted embodiment, and may use the cache contents to make adetermination as to whether to send the immediate renewal request or abatched renewal request. Independently of the lease renewal, the accessnode may initiate the requested operations on behalf of the client(e.g., using cached client session metadata and/or cachedlogical-block-to-physical-page mappings), and may provide theappropriate file store operation response 5663 to the client 5502.

In order to perform various types of file store operations at thedesired performance level, any of several approaches to the storage oflock state information for file store objects may be employed. FIGS. 57aand 57b illustrate alternative approaches to lock state management for asession-oriented file system protocol at a distributed storage service,according to at least some embodiments. In one approach, illustrated inFIG. 57a , the lock state indicators 5705 of a particular file systemmay be distributed among multiple extents. In some implementations ofthis approach, the LSIs containing lock and/or open state informationfor the various file store objects may be stored together with othertypes of metadata maintained for the entries, e.g., the correspondingnamespace DFS-DirectoryEntries (namespace entries), DFS-Inodes, and/orthe logical-block-to-physical-page mappings for the objects of the filesystem. Thus, for example, LSI 5705A for the root directory may bestored with other metadata 5704A for the root directory at one or morelogical blocks of a particular extent, LSI 5705B for directory D1 may bestored with other metadata 5704B for directory D1 at a different extent,and so on. Similarly, respective open/lock state information entries5705C, 5705D, 5705E and 5705F may each be stored in respective logicalblocks for directory D2, directory D3, file F1, and file F2. In thesecond approach, illustrated in FIG. 57b , the open/lock stateinformation for all the objects of a given file system may be stored ina consolidated fashion, e.g., within a single metadata extent 5754. Whenlooking up all the LSI entries for a given client session, e.g., forsession invalidation operation, multiple extents may have to be accessedif the distributed approach illustrated in FIG. 57a is used, while onlyone or a small number of extents may be required if the consolidatedapproach illustrated in FIG. 57b is used. However, under somecircumstances the consolidated approach may result in poorer resourceutilization than the distributed approach, e.g., because LSIs may bedeleted as the population of file store objects changes, and/or becausethe amount of storage eventually required for lock/open stateinformation for a given file system may not be easy to predict at thetime that the file system is created and the extent for its LSIs isobtained.

FIG. 58 is a flow diagram illustrating aspects of client sessionmetadata management operations that may be performed a distributedstorage service, according to at least some embodiments. As shown inelement 5801, a request to initialize or create a client session may bereceived from a client at an access subsystem node of a distributedstorage service that supports a stateful or session-oriented file systemprotocol such as NFS or SMB. In some implementations, an API requestingan explicit session initialization, similar to an NFS SetClientID API,may be used by the client. In other implementations, the request toestablish the session may be implicit, e.g., a session may beinitialized, if one does not already exist, in response to an open( )API invoked from the client. The session request may in someimplementations include an identification of the particular client(e.g., a value derived from an IP address and/or hostname of a host atwhich one or client processes are running) as well as a uniquesingle-use-only verifier value. If a client process exits and has to berestarted, or if the host or compute instance at which the clientprocesses run is rebooted, at least in some embodiments a new sessionmay have to be initialized, and a different verifier may be supplied tothe storage service in the corresponding session initialization request.

In the depicted embodiment, the metadata subsystem of the distributesstorage service may be responsible for storing client sessioninformation at persistent storage at one or more extents, while theaccess subsystem may be configured to cache session state information,e.g., in volatile memory and/or local persistent storage at the accessnode. In response to receiving the session request, the access node maytransmit a request for a session identifier, e.g., in an internalversion of the client's session request, to a selected metadata node(element 5804). The metadata node may be selected based on the client'sidentification information in some embodiments—e.g., in one embodimenttwo different metadata nodes MN1 and MN2 may be selected for respectiveclient sessions to be established for clients Cl1 and Cl2. The selectedmetadata node may allocate a logical block (mapped to some number ofphysical pages at metadata extents using one of the mapping techniquesdescribed earlier) for various elements of the client session metadatato be stored, including for example the lease settings for the session,the identity of the client, the identity of the responsible access nodefor the client session, and so on (element 5807). In at least someembodiments, a session identifier (e.g., NFS ClientID) may be determinedfor the new session based at least in part on the address at which thesession metadata is stored—e.g., a logical block address or a physicalpage address may be incorporated within, or used as, the sessionidentifier. The session identifier and an initial lease setting may beprovided from the metadata node to the access node (element 5810) in thedepicted embodiment. In some embodiments, only the session identifiermay be provided to the access node, and the access node may be able toretrieve other elements of the session metadata from the storagesubsystem using at least a portion of the session identifier as aparameter in a read request.

The session identifier and the lease information may be cached in asession metadata cache by the access node, and the session identifiermay be returned to the client (element 5813). The client may include thesession identifier as a parameter in subsequent file store operationrequests, e.g., in open( ), read( ), write( ), getattribute( ), orclose( ) calls directed at files or directories of the file system. Whenthe access node receives such an operation request, it may look up thesession information in its local cache, e.g., to verify that theclient's session is still open.

For some types of operations in the depicted embodiment, e.g., writeoperations directed to files, locks may be required in accordance withthe concurrency management techniques of the file system protocol inuse. Upon receiving a given file system operation request (comprisingthe session identifier), such as a write or a read directed to a filestore object F1, the access node may determine whether such a lock isneeded (element 5816). If a lock is needed and is not already cached atthe access node, a corresponding internal version of the operationrequest may be transmitted from the access node to a metadata node(element 5819). The metadata node may determine whether a conflictinglock state indicator already exists (e.g., because F1 is already lockedon behalf of another client). If such a conflicting lock is found (asdetermined in element 5820), the client's file system operation requestmay be rejected (element 5821), e.g., by sending an error messageindicating that the targeted object is already locked. If no conflict isfound, the metadata node may determine a persistent storage location fora logical block to be used to store state information for F1, includingfor example the corresponding lock state indicator (element 5822). Forexample, in some embodiments, one of the techniques illustrated in FIG.57a or 57 b may be used to allocate space for the lock state indicatorand/or other state metadata to be saved for F1. The state informationmay be stored at the persistent storage location (element 5825), and atleast a portion of the state metadata including the lock state indicatormay be provided to the access node.

The requested operation (e.g., the read or write directed to F1) may becompleted, e.g., as a result of an internal I/O request directed to thestorage subsystem by either the access node or the metadata node, and acorresponding response may be sent to the client. The access node mayadd the lock state indicator to its session metadata cache and use thecached lock state indicator, caches lease settings and/or the cachedsession identifier to respond to subsequent requests from the clientduring the session element 5828), e.g., without requiring interactionswith the metadata subsystem for at least some of the subsequentrequests. When and if the session expires, its metadata may be deletedfrom both the access node's cache and from the persistent storageallocated at the request of the metadata node (element 5831) in thedepicted embodiment. It is noted that in accordance with some filesystem protocols, at least a portion of the session-related metadata mayalso be provided to and/or cached at client-side components of theservice, e.g., daemons instantiated at the hosts at which applicationsutilizing the file storage service are run.

FIG. 59 is a flow diagram illustrating aspects of client session leaserenewal operations that may be performed a distributed storage service,according to at least some embodiments. As described earlier, a leasemay represent a time-bound grant of control of the state of a set offiles, directories or other client-accessible storage objects to aclient from storage service. As shown in element 5901, a file storeoperation request OR1 that belongs to a category of operations thatresult in automatic lease renewals may be received from a client Cl1 atan access node of the storage service during a client session CS1. Forexample, a read, write, open or close request directed towards aparticular file of a session-oriented file system such as NFS may bereceived. Different file system protocols may define respective sets ofleas-renewing operations in various embodiments. The remainingoperations illustrated in FIG. 59 may also be performed in response toan explicit lease renewal command in at least some embodiments. Therequest may include the client's session identifier (e.g., an NFSClientID), which may be usable as an index value for metadata records inthe access node's session metadata cache.

The access node may look up the lease information (e.g., when the leaseis set to expire) for the client session (element 5904), e.g., in thesession metadata cache. If the lease is due to expire within somethreshold time interval T (as determined in element 5907), the accessnode may transmit an immediate lease renewal request for CS1 to ametadata node (element 5913). If, however, the lease is due to expireafter the threshold time interval T, a lease renewal request for CS1 maybe added to a buffered set of pending lease renewal requests to be sentin a batch to the metadata node. If the operation request OR1 requiresstorage operations to be performed (e.g., if the request cannot besatisfied by data or metadata already cached at the access node), thestorage operations may be requested by the access node (element 5916),regardless of whether an immediate renewal request was sent or not. Inthe scenario where CS1's lease renewal request is buffered, one or moreof the buffered lease renewal requests may be transmitted to themetadata node asynchronously with respect to the operation request OR1(element 5919).

In at least some embodiments in which the buffering technique for leaserenewal requests is implemented, a different validity timeout may beconfigured or set for the version of the session metadata that is cachedat the access node (including for example the session identifier and theLSIs of the session) than is set for the persistent version of thesession metadata stored at the request of the metadata node. Forexample, in one implementation, if the lease timeout is set to 90seconds in accordance with the file system protocol settings, a validitytimeout of 120 seconds may be used for persistent session metadatarecords at the metadata subsystem, while a validity timeout of 30seconds (e.g., based at least in part on the difference between themetadata subsystem's validity timeout and the protocol's lease timeout)may be set for the corresponding records at the access node's cache.Using such different timeout combinations, at least some types ofpotential failures or delays at the access node may be accommodatedwithout causing clients to lose the benefits of their leasesprematurely. For example, with the example timeout settings introducedabove, since the access node would be required to refresh its cachedlease information once every 30 seconds from the metadata subsystem inany case, while the client's actual lease is valid for 90 seconds, abatching delay of a few seconds (e.g., a delay of less than 30 secondscaused by a failover of the access node to a replacement node) wouldtypically not be expected to result in any violations of the protocollease semantics. Since lease-renewing operations may be expected tooccur fairly frequently, the probability that the access node's shortervalidity timeout results in extra traffic between the access node andthe metadata subsystem may be kept quite low in such implementations. Itis noted that at least some of the techniques described earlier, such asthe use of conditional writes in read-modify-write sequences,distributed transactions, and/or replicated state machines in general,may also be used to manage client session-related metadata as well. Forexample, in one implementation, when a client session lease expires, anda plurality of session-associated lock state indicators distributedamong various nodes of the service have to be deleted, a distributedtransaction may be used.

Connection Balancing Using Attempt Counts

At some distributed storage systems expected to comprise thousands ofnodes and expected to handle tens or hundreds of thousands of concurrentclient requests, load balancing the client workload may be essential toachieving the targeted performance and resource utilization goals. In atleast some provider network environments, a collection of load balancingnodes may be established as the intermediaries between various servicesand the clients that wish to utilize the services. In some embodiments,such an intermediary load balancing layer may be established betweenclient devices and an access subsystem of a distributed storage service.Network connections (such as NFS mount connections) established onbehalf of clients to distributed storage services may typically befairly long-lived, and as a consequence the problems of workloadbalancing may become more complex than in environments in which usersessions are typically shorter (e.g., some types of web serverenvironments). A number of different techniques may be used to manageworkload levels of distributed storage service access nodes, including,for example, a connection balancing technique described below that takesinto account the number of unsuccessful attempts that have previouslybeen made to establish a connection on behalf of a particular client. Insome embodiments, connections may be voluntarily terminated by accessnodes under certain workload conditions, as also described below.

FIG. 60 illustrates a system in which a load balancer layer isconfigured for a distributed storage service, according to at least someembodiments. In the depicted embodiment, the load balancer layer 6090comprises a plurality of load balancer nodes (LBNs) 6070, such as nodes6070A, 6070B, and 6070C, implemented using resources of a providernetwork 6002. The access subsystem 6010 of the distributed storagesubsystem comprises a plurality of access node (AN) peer groups 6060,such as AN peer group 6060A comprising ANs 6012A, 6012B and 6012C, andAN peer group 6060B comprising ANs 6012K, 6012L and 6012M. The membersof an AN peer group may collaborate with each other for connectionrebalancing operations in at least some embodiments, as described belowin further detail. The members of an AN peer group 6060 may be selectedfrom among the plurality of access subsystem nodes of the storageservice based on any combination of a variety of criteria in differentembodiments—e.g., based on availability requirements of the accesssubsystem (e.g., such that a single localized power outage or otherinfrastructure outage does not cause failures at all the members of anAN group), latency requirements (e.g., such that different members ofthe group are able to support similar levels of latency), performancecapacity requirements (such that the total throughput that can behandled collectively by an AN peer group is above some desired minimum).In some implementations, an AN peer group may comprise a plurality ofaccess nodes that are all implemented on hardware servers mounted at asingle rack. In other implementations, AN peer group boundaries may notcoincide with rack boundaries; instead, other factors such as sharednetwork address prefixes, resilience-to-failure or the types/numbers offile stores being handled may be used to define peer groups.

In at least some embodiments, the TCP/IP (Transmission ControlProtocol/Internet Protocol) family of protocols may be used forcommunications between clients 180 and the storage service. A client 180may transmit, a connection establishment request to an LBN 6070 whosenetwork address (e.g., a virtual IP address) has been exposed as anendpoint for accessing the storage service. Various types of physical orvirtual networks 6022 may be used by the clients in differentembodiments. In one embodiment, as described earlier, some or all of theclients (such as compute instances configured as part of an isolatedvirtual network) may be instantiated at hosts within the providernetwork, and may thus use an internal network to connect to the loadbalancer nodes. In at least one embodiment, a load balancer node and aclient of the storage service may both execute at the same host (e.g.,as separate virtual machines), in which case no off-host networkconnection may be required. In another embodiment, a portion of anetwork external to the provider network 6002, such as a portion of theInternet may be used. In some embodiments, a plurality of LBNs may beconfigured to respond to traffic directed at a single IP addressassociated with the storage service. In one implementation, a particularLBN 6070 may first tentatively accept the client's connectionestablishment request, and that LBN 6070 may then attempt to establish acorresponding internal connection via network fabric 6024 (e.g., an L3network) of the provider network 6002 to an access node 6012. In atleast some embodiments, as described below, a given access node 6012 mayreject the internal connection request issued by the LBN under certainworkload conditions, and the LBN may consequently attempt to findanother access node 6012 that is willing to establish the internalconnection. In some embodiments, the specific criteria that an accessnode uses to accept or reject an LBN's request may depend on the numberof unsuccessful attempts that the LBN has already made—e.g., thecriteria may be relaxed as the number of unsuccessful attempts increase,so that the probability of connection establishment may increase withthe number of attempts.

In the depicted embodiment, each AN 6012 comprises two subcomponents: alocal load balancer module (LLBM) 6017 (e.g., LLBMs 6017A, 6017B, 6017C,6017K, 6017L and 6017M), and an access manager (AM) 6015 (e.g., AM6015A, 6015B, 6015C, 6015K, 6015L and 6015M). After a connection requesthas been accepted, in some embodiments an LLBM may be responsible forreceiving encapsulated TCP packets sent by an LBN on behalf of a clientover the network fabric 6024. In various implementations, the LBN mayencapsulate the client's TCP packets using a different protocol (e.g.,User Datagram Protocol (UDP) or some proprietary protocol usedinternally within the provider network), or using TCP itself—e.g., aclient's TCP packet (including its headers) may be included within anLBN TCP packet for the transmittal between the LBN and the LLBM. TheLLBM may unpack or de-capsulate the packets before passing the packetson to a TCP processing stack associated with the local AM. In someimplementations the LLBM may change contents of one or more clientpacket headers such as the TCP sequence number before the transfer tothe TCP processing stack. In at least some embodiments, themanipulations of the client packets (e.g., encapsulation/unpacking,changing headers, etc.) by the combination of the LBN and the LLBM maymake it appear to the TCP processing stack as though the packet wasreceived on a TCP connection established directly with the client 180rather than via the LBN and the LLBM. The AM 6015 may implement storageservice front-end logic, including, for example, caching metadata,managing interactions with the metadata subsystem 120 and/or the storagesubsystem 130, and so on. In addition, in some embodiments, the AM 6015may collect a set of local workload metrics of various resources of theAN, such as CPU metrics, network metrics, memory metrics and the like,that can be used for decisions on accepting additional connections. Inone embodiment, the AMs of different peers of a peer group 6060 mayquery each other regarding their workload levels as described in greaterdetail below.

According to at least some embodiments, a connection request comprisingan attempt count parameter may be received at an access node 6012 froman LBN 6070 on behalf of a client 180. The attempt count parameter mayindicate the number of times the load balancer component has attemptedto establish a connection on behalf of that particular client 180. Inone embodiment, a client may submit a request to mount a file system(e.g., and NFS mount command), and the LBN may generate its connectionrequest in response to receiving the mount command; the connectionestablished as a result may be termed a “mount connection” and may beused for several subsequent requests from the same client. In otherembodiments, other storage service commands or requests (i.e., requestsother than mount requests) may also or instead trigger connectionestablishment requests. Upon receiving the connection request, the ANmay identify one or more workload threshold levels (e.g., respectivethreshold levels Th1, Th2, . . . for a plurality of resources) to beused for an acceptance decision regarding the connection request. Atleast one of the threshold levels may be based on the attempt countparameter in some embodiments—e.g., for the first attempt, the CPUworkload threshold may be Tc, while for a second attempt, the CPUworkload level may be set to (Tc+delta), making it more likely that theconnection is accepted on the second attempt. In one example scenario,if threshold level Tc is identified for CPU workload, and thresholdlevel Tn is identified for network workload, the connection may beaccepted if a CPU workload metric of the AN is below Tc and a networkworkload metric is below Tn. In another scenario, the connection may beaccepted if either the CPU workload metric or the network workloadmetric is below the corresponding threshold. The workload metrics usedfor comparison with the thresholds may be computed over some timeinterval in some embodiments as discussed below, e.g., in order toreduce the impact of short-term workload fluctuations on the connectionacceptance decision.

In response to a determination that the local workload metric or metricsof the access subsystem node are below the corresponding workloadthreshold levels, an indication that the connection is accepted may beprovided to the requesting LBN 6070. Both the connection request and theacceptance indication may be formatted in accordance with the particularprotocol being used for communication between the LBNs and the LLBMs(e.g., UDP, TCP, or some other protocol). The LBN 6070 may in someembodiments confirm to the client that the connection has been acceptedby the AN. If the AN 6012 selected by the LBN cannot accept theconnection (e.g., if the local workload metrics are above the thresholdidentified), a connection rejection message may be sent to the LBN. TheLBN may then transmit its request (with the attempt count parameterincremented) to another AN, and this process may be repeated asillustrated in FIG. 61 and described below, until either the connectionis successfully established or the number of attempts exceeds somemaximum number of attempts permitted.

After a connection is successfully established, when the LBN 6070receives a client-generated packet indicative of a storage servicerequest, the LBN may transmit the packet to the LLBM at the accesssubsystem node (e.g., in an encapsulated format). The LLBM maymanipulate the contents of the message received from the LBN (e.g., tounpack the original client-generated packet), and pass the originalpacket on to the AM 6015 for processing. Depending on the nature of theoperations that have to be performed in response to the storage request,the AM may in some cases have to contact either the metadata subsystem120, the storage subsystem 130, or both back-end subsystems. Anindication of the storage service request may be transmitted to theappropriate subsystem(s). If the client's service request requires aresponse, the response may flow in the opposite direction—e.g., from theback-end subsystem(s) to the AN, from the AN to the client via the LBN.In at least some embodiments in which incoming packets are encapsulatedby the LBN and unpacked by the LLBM, the LLBM may similarly encapsulateoutgoing packets and the LBN may unpack the packets before passing themon to the client 180.

FIG. 61 illustrates example interactions between a load balancer nodeand a plurality of access subsystem nodes of a distributed storageservice, according to at least some embodiments. In the depictedembodiment, a virtual IP address 6105 (e.g., an IP address that can bedynamically associated with different network interfaces, e.g., atdifferent compute instances of a provider network's virtual computingservice, and is not tied to a single network interface) may be exposedto enable clients to submit connection requests and other storageservice requests to the storage service. One or more LBNs 6070 may beresponsible for accepting traffic directed at the virtual IP address atany given time. In at least some embodiments, the LBNs (and/or the ANs)may be implemented using compute instances—e.g., a given LBN maycomprise a process executing at a compute instance of a providernetwork's virtual computing service, launched at a commodity hardwareserver. The client may submit a connection establishment request 6108 tothe virtual IP address 6108.

In the depicted embodiment, the LBN 6070 may receive the client'srequest, and select a particular AN 6012B as the first AN to which itshould send a corresponding internal connection request. A number ofdifferent techniques may be used to select the AN—e.g., random selectionmay be used in some embodiments, round-robin selection may be used inother embodiments, and so on. In some embodiments, each LBN may beaffiliated with a set of ANs (such as one or more AN peer groups definedbased on availability, latency, capacity, or other criteria mentionedearlier), and the LBN may cycle through its affiliated ANs in adesignated order for its connection attempts. In some embodiments, somenumber of the LBNs and some number of the ANs may both be located at thesame rack, and an LBN may select an AN from within its own rack first.The LBN may submit the first connection attempt 6132A to an LLBM 6017Bat the selected AN 6012B, e.g. with the attempt count parameter set to 1in the depicted embodiment. (The attempt count parameter may be set tozero for the first attempt in some implementations.) The decisionregarding acceptance or rejection of the request may be made either bythe AM 6015 at the targeted AN, by the LLBM at the targeted AN, or bythe combination of the LLBM and the AM at the targeted AN, in differentembodiments.

If the first AN contacted sends a rejection 61234A to the LBN (e.g.,based at least in part on one or more local workload metrics 6115Bexceeding corresponding thresholds), the LBN may select a second AN (AN6012A in the depicted example). The LBN 6070 may submit a secondconnection request attempt 6132B, with an incremented attempt countparameter, to the LLBM 6017A at the second AN. If a rejection 6134B isreceived again (e.g., based on AN 6012A's local workload metrics 6115A),the LBN 6070 may select a third AN 6012C, and submit the third attempt6132C to its LLBM 6017C. In the depicted example scenario, the third AN6012C sends back an acceptance 6136 based on an analysis of its localworkload metrics 6115C, and the connection is established accordinglybetween the AM 6015C and the client 180. After the successfulestablishment of the connection, network packets between the storageservice and the client 180 flow along path 6157 in the depictedembodiment. For example, the client may send a packet to the LBN 6070,the LBN may send the packet (potentially using an encapsulated ormodified representation) to the LLBM 6017C, a packet manipulator 6155 ofthe LLBM may unpack or modify the received packet, and send the outputof the manipulation to the AM 6015C. AM 6015C may then initiate thestorage operations required, which may involve interactions with themetadata and/or storage subsystems.

FIG. 62 illustrates examples of connection acceptance criteria that mayvary with the number of connection attempts made, according to at leastsome embodiments. In the depicted embodiment, for a given resource, thenative or baseline capacity 6202 of an AN with respect to that resource(such as CPU or network bandwidth) may be modified by a failure overheadfactor 6204 to arrive at an adjusted capacity (AC) 6206 to be used forconnection acceptance decisions. For example, if the native CPUcapability of the AN is X operations per second, in one scenario, onefifth of that capacity (0.2X) may be set aside to compensate fortemporary workload increases that might occur in the event of failuresof various kinds. Thus, the adjusted CPU capacity would be set to 0.8X(X−0.2X) operations per second in such a scenario.

The local workload metrics collected for a given resource at an AN mayexhibit short-term variations as well as long-term trends. Since theconnections established for storage service operations (such as mountconnections set up for NFS) may typically be long-lasting, it may not beadvisable to accept/reject the connections on the basis of just the mostrecent metrics alone. Accordingly, an adjusted load metric (AL) 6216 maybe obtained from a combination of the most recent metric 6212 and someset of historical metrics 6214 (e.g., metrics collected for thatresource over the last 15 minutes or an hour). In some embodiments, adecay function 6215 (e.g., an exponential decay or a linear decay) maybe applied to historical metrics when computing the adjusted load, e.g.,to represent or model the reduction in the importance of the metricsover time.

To accept a connection request with a specified attempt count parameterat an AN, the adjusted load 6216 for a given resource may be compared toa threshold (expressed in terms of the adjusted capacity for thatresource) that is dependent on the attempt count. Thus, as indicated inthe connection acceptance criteria table 6255, a connection request withan attempt count parameter equal to one may be accepted if the AL forthe resource being considered is less than or equal to 0.5*AC. If theconnection request has failed once, and the attempt count is accordinglyset to 2, the connection may be accepted of the AL is no greater than0.55*AC. For an attempt count value of 3, the acceptance criterion maybe relaxed further so that the connection is accepted if AL is nogreater than 0.6*AC; for attempt count=4, AL may have to be no greaterthan 0.75*AC, and for attempt count 5, AL may have to be no greater than0.85*AC. Thus, the more times that a connection is rejected in thedepicted embodiment, the more heavily loaded the AN that eventuallyaccepts it may be allowed to be. In other embodiments, the oppositeapproach may be used, in which in order to accept a connection requestwith an attempt count K, the workload level of the accepting node mayhave to be lower than the workload level required to accept theconnection request with a lower attempt count (K-L). Such an approach,in which the relative ease of acceptance of a connection decreases asthe attempt count increases, may be used for example in a scenario inwhich new connection attempts are to be discouraged under heavy loadconditions. The threshold conditions, as well as the parameters andfunctions (e.g., the decay function) used for the computation of the ACand the AL, may all be configurable settings in at least someembodiments. The number of distinct attempt count values for whichacceptance criteria are defined may vary in different embodiments, andmay itself be a configurable parameter in at least one embodiment. Insome embodiments, the parameters, functions and/or thresholds may bedynamically modified over time, e.g., based on an analysis of theresults achieved. In at least some embodiments, some of the acceptancecriteria may be the same for a range of attempt count values—e.g., forattempt counts 1 and 2, the same threshold value may be used.

In some embodiments, as mentioned above, local workload levelsassociated with more than one resource may be taken into account whenmaking connection acceptance decisions. FIG. 63 illustrates examples ofconnection acceptance criteria that may be dependent on workload levelsassociated with a plurality of resources, as well as on connectionestablishment attempt counts, according to at least some embodiments.Five examples of adjusted load levels and corresponding adjustedcapacities are shown in array 6312. AL[CPU] represents the adjusted CPUworkload of the access node, while AC[CPU] represents the adjusted CPUcapacity. AL[Net] represents adjusted network load, and AC[Net]represents adjusted network capacity. AL[Mem] represents adjusted memoryload, and AC[Mem] represents adjusted memory capacity. AL[Dsk]represents adjusted local storage device capacity load at the accessnode, and AC[Dsk] represents adjusted storage device capacity. In atleast some embodiments, adjusted loads and capacities may also bedetermined for logical resources such as open sockets that arerepresented by operating system structures at the access nodes. Theadjusted workloads (AL[OSS]) and the adjusted capacities (AC[OSS]) forsuch operating system structures may be considered in connectionacceptance decisions in at least some embodiments. For each resource,the adjusted load and the adjusted capacity may be expressed in the sameunits—e.g., if the network load is expressed in packets/second, thenetwork capacity may also be expressed in packets/second.

Thresholds expressed in terms of the AC array elements may be determinedfor each of various attempt count values, as indicated in multi-resourceconnection acceptance criteria table 6355. Different combinations ofresources may be taken into account for different attempt count levelsin the depicted embodiment—e.g., for attempt count=2, thresholds forCPU, network, and memory may be compared to the corresponding adjustedloads, while for attempt count=K, only CPU loads and thresholds may becompared. The “&&” symbols in table 6355 indicate Boolean “AND” s, sothat, for example, at attempt count=4, both the CPU and network criteriamay have to be met to accept a connection. In various embodiments,different Boolean combinations of the load vs. threshold comparisons fordifferent resources may be used—e.g., either ORs, ANDS, or both ORs andANDS may be used.

FIG. 64 is a flow diagram illustrating aspects of operations that may beperformed to implement connection balancing based on attempt counts at adistributed storage service, according to at least some embodiments. Asshown in element 6401, a set of load balancer nodes' network addresses(e.g., virtual IP addresses that may be accessible from within anisolated virtual network of the type illustrated in FIG. 3) may beexposed to clients to enable them to submit storage-related requests tothe service. A connection request from a client may be received at aparticular LBN, LBN1 (element 6404). LBN1 may in turn submit acorresponding connection request, comprising an attempt count parameterindicating the number of times an attempt to establish the connectionhas been made, to a selected access node AN (element 6407). Variousapproaches may be used to selecting the next AN to which a connectionestablishment attempt is directed—e.g., the ANs may be selected atrandom, using a round-robin approach, or based on some other factorssuch as how recently a connection was established at the AN from LBN1.

The AN may determine adjusted local workload metrics (WM) for one ormore resources, and the threshold values (WT) with which those workloadmetrics are to be compared to accept/reject the connection (element6410). At least some of the thresholds may differ for different attemptcount values. The thresholds may be expressed in terms of adjustedresource capacities in some embodiments, and the adjusted resourcecapacities may in turn derived from native or baseline resourcecapacities and failure adjustment factors. In some embodiments, variousBoolean combinations of resource-specific acceptance conditions may beused, as indicated in FIG. 63. If the acceptance criteria are met, e.g.,if WM<=WT for the resources being considered for the attempt countvalue, as determined in element 6413, LBN1 may be informed that theconnection has been accepted (element 6428). After the connection isaccepted, a packet representing a storage request may be received atLBN1 from the client and transmitted to an LLBM (local load balancermodule) at the AN to which the connection was established (element6431). In some implementations, the client's packets may be encapsulatedby LBN1, and unpacked or extracted by the LLBM (element 6434). The LLBMmay transfer the packet to a network processing stack at the AN, wherethe packet contents may be analyzed to determine which storage serviceoperations are needed to respond to the client's request. Requests forthose operations may be sent to other subsystems of the service asneeded (e.g., to the metadata subsystem and/or the storage subsystem)(element 6437).

If the criteria for accepting the connection are not met at the ANselected by LBN1 (as also detected in element 6413), the connectionattempt may be rejected (element 6417). If LBN1 has already made themaximum number of attempts permitted (“Max-attempt-count”) to establishthe connection (as detected in element 6419), an error message may bereturned to the client in some embodiments (element 6422) indicatingthat connection establishment failed. In many embodiments, theattempt-count-based acceptance criteria may be selected in such a waythat the likelihood of failure to establish a connection is kept verylow. The number of connection establishment failures may be tracked, andadditional ANs may be configured as needed to keep the number orfraction of failures below a target level.

If LBN1 has not yet submitted the maximum permissible number ofconnection attempts for the client (as also detected in element 6419),LBN1 may select another AN to which a connection request should besubmitted (element 6425). A new connection attempt, with the attemptcount parameter incremented, may be sent to the selected AN, and theoperations corresponding to elements 6407 onwards may be repeated. Insome embodiments, the same kinds of techniques that were used by LBN1 toselect the first AN may be used for selecting ANs for subsequentattempts. In other embodiments, LBN1 may change its criteria forselecting ANs based on attempt count—e.g., the first AN may be selectedat random, while the next AN may be selected based on how successfulLBN1 has been in previous attempts at connection establishment withvarious ANs. In one such embodiment, an LBN may maintain statistics onits connection establishment success rate with various ANs, and may usethe statistics to select ANs that have been able to accept connectionsmore frequently in the past.

Connection Re-Balancing Using Peer Group Workload Information

Connections established to file storage systems, such as NFS mountconnections, may often persist for a long time. Information that wasrelevant to the connection acceptance decision at the time theconnection request was received, such as the resource workload levels ofone or more resources during some prior time interval, may notnecessarily be indicative of current conditions at the access node atsome later point during the connection's lifetime. In one example, anaccess node may have accepted a connection at a time when its adjustedCPU load was X, but the connection may still be in use at a later timewhen the adjusted CPU load has remained at 1.5X for some period.Accordingly, in some embodiments access nodes may attempt to re-balancetheir workloads under some circumstances.

FIG. 65 illustrates an example of an access subsystem of a distributedstorage service at which client connection re-balancing may be attemptedbased on workload indicators of members of a peer group of access nodes,according to at least some embodiments. An access node peer groupcomprising three nodes, ANs 6512A, 6512B and 6512C is shown. Membershipin a peer group may be determined based on a variety of factors indifferent embodiments as mentioned above, including for exampleavailability, latency, capacity, co-location, or shared network addressprefixes. In the depicted embodiment, each peer group member may collectat least two types of workload metrics: local workload metrics 6155(e.g., 6115A, 6115B or 6115C) such as the observed loads discussedearlier for CPUs, network, memory and other resources of the AN, andindicators 6502 of the workload levels at other ANs of the peer group.In the depicted example configuration, AN 6512A may collect peerworkload indicators 6502A from ANs 6512B and 6512C, AN 6512B may collectpeer workload indicators 6502B from ANs 6512A and 6512C, and AN 6512Cmay collect peer workload indicators from ANs 6512A and 6512B. Themanner in which the workload indicators are collected, and/or the natureor contents of the workload indicators, may differ in differentembodiments. In some embodiments, for example, a given AN may simplysend a connection establishment query to each of its peers at someselected points in time, and receive a response indicating whether thepeer is willing to accept a connection or not. In some embodiments inwhich connection acceptance decisions may be affected by attempt countparameters as discussed earlier, the connection establishment queriesmay also include an attempt count parameter (e.g., an attempt countparameter value of “1” may be used). The AN that sends the queries maykeep track of how many connections each of the peers was willing toaccept during some time interval. In embodiments in which each AN isexpected to take its local workload metrics into account when makingconnection acceptance decisions, the connection acceptance rate mayserve as an accurate and easy-to-obtain workload indicator. In otherembodiments, the ANs may simply exchange digests or summaries of theirlocal workload metrics periodically or according to some schedule, andsuch summaries may be used as workload indicators. In some embodiments,workload indicators may be sent only in response to queries, while inother embodiments, workload indicators may be pushed to a peer groupmember regardless of whether a query was received or not. The specifictechnique used for sharing workload information may be selected (ormodified) in the depicted embodiment such that the total traffic andprocessing overhead associated with queries/responses 6570 is kept belowa threshold.

Each AN of the peer group has some set of established or openconnections, such as connections C11, C12, . . . C1 n at AN 6512A,connections C21, C22, . . . C2 p at AN 6512B, and connections C31, C32,C3 n at AN 6512C. The access nodes may each maintain respectiveconnection statistics 6504 on their open connections—e.g., statistics6504A may be maintained at AN 6512A, statistics 6504B may be maintainedat AN 6512B, and statistics 6504C may be maintained at AN 6512C.Connection statistics 6504 maintained for a particular connection Cjkmay include, for example, a measure of the age of the connections (e.g.,when Cjk was established), the amount and time distribution of trafficon the connection, the number of storage operations (e.g., file opens,reads, writes, etc.) that have been requested on the connection, thesizes of the packets, the number of packets dropped, and so on. If andwhen an AN determines that a connection is to be closed or disconnectedfor workload rebalancing, the connection statistics 6504 may beanalyzed, and one or more connections may be closed in accordance with aclosure target selection criterion that may be based on the statistics.Depending on the network protocol in use, the AN may send theappropriate messages to initiate the disconnection to the client; insome embodiments, an exchange of messages may be required to cleanlyclose the connection.

In some embodiments, a decision to close a connection may be made at anaccess node 6512 if both of the following conditions are met: (a) atleast one local workload metric 6115 at that access node exceeds arebalancing threshold and (b) a peer capacity availability criterionderived from the collected workload indicators is met. For example, inone scenario, if at least 70% of the peers of an AN 6512 would bewilling to accept a new connection based on the latest availableworkload indicators, and AN 6512's own workload level has reached a highenough level, AN 6512 may decide to close or drop a selected connection.The local workload-based criterion may be used so that connectionrebalance are only attempted when the AN's local resources are heavilyutilized (e.g., so heavily utilized that no new connection would beaccepted). The peer capacity availability criterion may be taken intoaccount so that, for example, the client at the other end of a closedconnection would have a reasonable chance of establishing a connectionand continuing its storage service request stream.

If a decision to close some connection (or a plurality of connections)is made, in at least some embodiments the particular connection(s) to beclosed may be selected based on an analysis of the connection statistics6504 as mentioned earlier. For example, in order to avoid oscillationscenarios in which the same client's connections are closed repeatedlyat different ANs, connections that have been in existence for longerthan some threshold time may be preferred as closure targets. In someembodiments, connections whose traffic has led to greater resource use(e.g., connections that have been used for resource intensive storageoperations) may be considered preferred targets for closure, relative tothose connections that have led to more modest resource utilization atthe AN. The AN may then initiate the closure of the selectedconnection(s) in accordance with the particular network protocol (e.g.,TCP) that is being used. In response to the closure of the connection,the client may try to establish another connection in at least someembodiments. A load balancer node (which may be the same LBN as the onethat participated in the establishment of the now-closed connection, ora different LBN) may then issue a connection establishment request inbehalf of the client to a selected AN (e.g., belonging to the peer groupof the AN that closed the connection). A connection establishmentprotocol similar to that described earlier may be used until an ANwilling to accept the client's connection is found (or until the loadbalancer reaches the maximum attempt count). If the peer capacityavailability criterion used to make the connection rebalancing decisionis a good indicator of the willingness of ANs to accept connections, theclient may soon be able to establish a new connection to replace theclosed connection. In at least some embodiments in which asession-oriented file system is supported, it may even be possible forthe client to continue with the same session that was being used beforethe connection rebalancing, as described below with reference to FIG.68. In one embodiment, after a particular AN has closed a connectionwith a particular client C1, if the AN receives a subsequent connectionrequest on behalf of the same client C1 within a re-connection thresholdtime interval, the connection request may be rejected, e.g., so as toavoid scenarios in which the same client has its connections closedrepeatedly.

In one embodiment, a load balancer node may be able to establish areplacement connection transparently with respect to the client—e.g.,without the client being informed or made aware that a closing of itsconnection was initiated by an AN. The load balancer node may be able todetect (e.g., by examining packet headers and/or packet body contentsreceived from the AN) that a rebalancing-related disconnection has beeninitiated. Upon discovering this, the load balancer node may select adifferent AN, and initiate establishment a different connection to thedifferent AN without informing or notifying the client. If the loadbalancer node is able to find an AN that accepts its request, in atleast some embodiments, from the client's perspective nothing may appearto have changed (i.e., no effects of the re-balancing may be noticed bythe client). In order to achieve such transparency, in someimplementations the load balancer and the access subsystem maycollectively have to manage connection state information transferbetween the AN that initiated the disconnection and the replacement AN.

FIG. 66 illustrates an example of connection acceptance and re-balancingcriteria that may be used at an access subsystem node, according to atleast some embodiments. In the depicted embodiment, attempt-count basedconnection acceptance thresholds may be used, in a manner similar tothat described earlier. However, it is noted that in at least someembodiments, the connection rebalancing technique used may be orthogonalto the connection acceptance criteria—e.g., connection rebalancing maybe used in an embodiment even if the attempt-count based connectionacceptance techniques described above are not used.

In the embodiment depicted in FIG. 66, as in some of the examplesdiscussed earlier, the threshold used for different attempt count levelsmay make it easier for a connection to be accepted as the attempt countvalue rises. Thus, for example, to reject a connection request withattempt count equal to three, an AN's adjusted CPU load (AL[CPU]) wouldhave to exceed 0.6 times the adjusted CPU capacity (AC[CPU]) and theAN's adjusted network load (AL[net]) would have to exceed 0.6 times theadjusted network capacity (AC[net]). However, to reject a connectionrequest with an attempt count value of four, the adjusted loads for CPUand network would each have to be higher (0.8 times AC[CPU] and 0.8times AC[net], respectively).

A combination of several factors contributes to the example rebalancingcriteria illustrated in FIG. 66. First, the adjusted local load levelsfor the CPU, the network, or both, must exceed 0.85 times thecorresponding adjusted capacity. Second, the adjusted memory load mustexceed 0.85 times the adjusted memory capacity. Third, at least 600seconds must have elapsed since the previous connection was closed atthe access node due to rebalancing. And fourth, the estimatedprobability that a peer access node would be willing to accept a newconnection (which may be obtained from the workload indicators collectedfrom peer group members) may have to exceed 70%. Thus, a fairlystringent set of tests may have to be passed before a connection isterminated by an AN in the depicted embodiment.

FIG. 67 is a flow diagram illustrating aspects of operations that may beperformed at an access subsystem of a distributed storage service toimplement connection re-balancing, according to at least someembodiments. As shown in element 6701, a number of network connectionsC1, C2, . . . , Cn may be established between an access node AN1 of amulti-tenant distributed storage subsystem and one or more load balancernodes (LBNs) on behalf of one or more clients of the service. Asdescribed earlier, in some embodiments a set of network addresses (e.g.,private virtual IP addresses accessible from within an isolated virtualnetwork of a provider network, or public accessible IP addressesaccessible from the Internet) may be configured for the load balancersand exposed to the clients that wish to access the service. In someembodiments, attempt-count based connection acceptance criteria may havebeen used to set up the connections C1-Cn, while in other embodimentsthe connections may have been established without taking attempt countsinto consideration. In some embodiments, AN1 may comprise a local loadbalancer module (LLBM) that intercepts and manipulates packets sent byLBNs as described earlier, while in other embodiments AN1 may notinclude such LLBMs.

During some time period T, AN1 may collect two kinds of workloadinformation (element 6704): local workload information pertaining toresources such as AN's CPU(s), AN's networking modules, and the like,and peer group workload indicators obtained from a number of peer ANs.In some embodiments, AN1 may submit workload-related queries to aselected set of peers (e.g., members of a peer group selected based onthe kinds of criteria mentioned earlier), and the workload indicatorsmay be received in response; in other embodiments, the ANs of a peergroup may proactively push their workload indicators to each other atvarious points in time. In some implementations, AN1 may submit a queryto a peer AN (e.g., AN-k) from time to time to determine whether AN-k iswilling to accept a connection, and AN-k's response may be considered anindicator of AN-k's workload. In at least one implementation, AN1 maysend a connection establishment request to AN-k (e.g., instead ofsending a query about connection establishment). In some embodiments, anAN may provide a digest or summary of its current local workloadestimates periodically to peer ANs, either on demand or proactively. Inone embodiment, the workload indicators may be piggybacked on othertypes of messages exchanged between the ANs, e.g., on administrativemessages or heartbeat messages.

Several criteria may have to be met before a connection is selected fortermination or closure in the depicted embodiment. AN1 may determinewhether its local workload metrics exceed a first re-balancing threshold(element 6707). The local workload metrics may be expressed usingadjusted values that take the variation of the raw metrics over timeinto account in some embodiments, as described earlier with respect toadjusted load (AL) calculations for connection acceptance. The firstre-balancing threshold may be expressed in adjusted capacity units forvarious resources in some embodiments, which set aside some of thenative resource capacity as overhead for dealing with possible failures,as also described earlier with respect to adjusted capacities (ACs) usedfor defining connection acceptance criteria. In other embodiments,different sets of workload metrics and/or resources may be taken intoaccount for re-balancing decisions than are considered for connectionacceptance decisions.

If the local workload-based criterion for re-balancing is met, AN1 maydetermine whether a peer capacity availability criterion has been met(element 6710). The peer capacity availability criterion may bedetermined based on the workload indicators obtained from the other ANsin the depicted embodiment. In at least some embodiments, meeting thepeer availability criterion may indicate that there is a reasonably highprobability that if AN1 terminates a connection to a particular client,that client would be able to establish a connection with another AN. Forexample, in one scenario the peer capacity availability criterion may bemet if AN1's own adjusted loads (for some set of selected resources)exceed 90% of the corresponding adjusted capacities, while AN1 candetermine using peer workload indicators that at least 75% of themembers of its peer group have adjusted loads of less than 40% of thecorresponding adjusted capacities and would therefore be likely toaccept new connections. It is noted that at least in some embodiments,the most recent workload indicator available at AN1 for a given peerAN-k may represent AN-k's state as of some previous point in time, andthat different workload indicators may represent different points intime. In such embodiments, the peer capacity availability determinationmay therefore be based on approximate rather than exact data.

If the local workload criterion for re-balancing and the peer capacityavailability criteria are met, in the depicted embodiment AN1 may alsodetermine whether any of its connections were closed for re-balancingpurposes within the last Tmin units of time (element 6713). For example,in the scenario illustrated in FIG. 66, Tmin was set to 600 seconds. Iftime greater than the minimum threshold setting Tmin has expired since aprevious rebalancing-related connection termination (or if this is thefirst re-balancing being attempted at AN1), a particular connection Cjmay be chosen for termination (element 6716) based on a closure targetselection policy. The target selection policy may take various factorsinto account such as the age of the connection (connections that weremore recently established may be less likely to be selected in someembodiments to avoid oscillating behavior), the amount of traffic on theconnection, the amount of usage of various AN resources (e.g., CPU,memory, etc.) associated with the connection, and so on. In someembodiments AN1 may utilize the connection statistics 6504 to select aclosure target.

The termination or closing of the selected target connection may beinitiated from AN1 in the depicted embodiment (element 6719), e.g., inaccordance with the appropriate connection termination syntax of thenetworking protocol in use. Upon determining that the connection hasbeen dropped/closed, the client on whose behalf Cj was established maysubmit another connection establishment request to a selected LBN(element 6722). The LBN may accordingly establish a connection, e.g.,with some other AN, e.g., AN2 on behalf of the client (element 6725). Itis noted that, depending on the connection acceptance criteria in useand on the changes in AN1's workload, this new connection may in somesituations be accepted by AN1 itself.

In the embodiment depicted in FIG. 67, if the local workload-basedrebalancing threshold is not met (as detected in element 6707), AN1 maycontinue its regular operations, collecting local and peer workloadinformation for subsequent time periods as indicated in element 6704. Ifone of the other two conditions for re-balancing are not met—e.g., ifthe peer capacity availability criterion is not met (element 6710) orinsufficient time has elapsed since the last connection was terminatedfor re-balancing—AN1 may take some additional actions in the depictedembodiment to deal with its excessive workload. For example, as shown inelement 6728, AN1 may optionally start throttling one or more of itsopen connections, e.g., by delaying the processing of selected packets,or by dropping packets. Of course, depending on the nature of thenetworking protocol in use, such actions may in some cases lead toretransmissions from the client, and may not be of much immediate help,at least until enough time elapses that a connection can be selected fortermination. In another embodiment, if the local workload-basedrebalancing threshold of element 6707 is met, AN1 may close a selectedconnection even if at least one of the other two conditions(corresponding to elements 6710 and 6713) is not met. It is noted thatthe three conditions that are considered to determine whether to close aconnection in FIG. 67 may be checked in a different order than thatshown in some embodiments, e.g., in some embodiments it may be the casethat the time that has elapsed since the previous termination may bechecked first, or that the peer capacity availability may be checkedfirst.

In some embodiments, at least one of the file system protocols supportedat a distributed storage service may be session-oriented as describedearlier, e.g., session identifiers may be generated for clients andassociated with resource leases and/or locks. The termination of aclient connection for rebalancing may result in undesired sessiontermination in such embodiments unless proactive preventive steps aretaken. FIG. 68 is a flow diagram illustrating aspects of operations thatmay be performed at a distributed storage service to preserve clientsessions across connection re-balancing events, according to at leastsome embodiments. When a client session CS1 is established for a clientCl1, e.g., in response to an explicit session establishment request orwhen the client Cl1 issues a particular type of storage request,corresponding session metadata may be stored by or at a metadatasubsystem node of the service which receives the session establishmentrequest from a particular AN. As shown in element 6801, that sessionmetadata may include a field identifying the particular access node thatis being used for CS1 (e.g., the AN that submitted the sessionestablishment request to the metadata node and is intended to be usedfor subsequent storage requests from Cl1). As also illustrated in FIG.55, such a field may be referred to as the “responsible access node”(RAN) field. The client Cl1 may specify a session identifier (e.g., anNFS “ClientID” parameter) that is generated as part of the sessionmetadata in its subsequent storage-related requests sent via AN1.

As shown in element 6804, AN1 may subsequently determine that Cl1'sconnection is to be terminated/closed for rebalancing, e.g., using thekinds of re-balancing criteria discussed above. Accordingly, the RANfield of the session metadata may be set to “null” (or to some othervalue indicating that no AN is responsible) (element 6807). The changeto the metadata may be performed by the metadata node at the request ofAN1 in some embodiments. The connection may be terminated at theinitiative of AN1.

Eventually, after Cl1 realizes that the connection is closed, Cl1 maysend another request, e.g., to a load balancer node, to try tore-establish connectivity to the storage service (element 6810). Adifferent access node (AN2) may respond to the connection establishmentrequest submitted on behalf of Cl1 by the LBN to accept the connection(element 6813). Client Cl1 may submit a storage service request (e.g.,an open( ), read( ) or write( )) with the same session identifier thatit was using prior to the connection's termination (element 6816). AN2may receive such a storage service request, and send a query to themetadata subsystem to determine the status of the metadata correspondingto the client-specified session identifier (element 6819). If themetadata subsystem is able to find session metadata for the specifiedsession identifier, and if the RAN field of that metadata is set to“null” (as detected in element 6822), this may indicate to AN2 that itis acceptable for AN2 to continue CL1's session with the existingmetadata, and to assume responsibility for Cl1's session. Accordingly,the RAN field of CS1's metadata may be set to AN2's identifier (element6825) and CS1 may be resumed. Otherwise, if for some reason CS1'smetadata records are not found, or if the RAN field in CS1's metadatawas not set to “null”, a new session may be created for the client(element 6828) in the depicted embodiment. Establishing the new sessionmay involve the acquisition of one or more locks/leases in at least someembodiments, and may in such embodiments require more resources than ifthe current session could be resumed with AN2 as the responsible accessnode.

It is noted that in various embodiments, operations other than thoseillustrated in the flow diagrams of FIGS. 8a, 8b , 9, 10, 15, 20, 21,22, 23, 27, 28, 32, 38, 41, 42, 43, 44, 51, 52, 53, 58, 59, 64, 67 and68 may be used to implement the distributed file storage servicetechniques described above. Some of the operations shown may not beimplemented in some embodiments, or may be implemented in a differentorder, or in parallel rather than sequentially. In at least someembodiments, the techniques described above may be used for managingworkload variations at other types of storage services than filestores—e.g., similar techniques may be used for storage devices thatexpose volume-level block storage interfaces, unstructured storagedevices that allow arbitrary storage objects to be accessed using webservice interfaces rather than file system interfaces, or for accessingtables or partitions of relational or non-relational databases.

Use Cases

The techniques described above, of implementing highly scalable,available and durable file storage systems that support one or moreindustry-standard file system interfaces may be useful in a number ofscenarios and for a variety of customers. Many customers of providernetworks have already migrated several of their applications to thecloud to take advantage of the enormous amount of computing power thatcan be harnessed. However, several constraints may remain for suchapplications with respect to the ability to store very large amounts ofdata (e.g., petabytes) within a single file, and then to access the filefrom large numbers of clients concurrently without impactingperformance. Scalability constraints may also remain with respect tofile system directory hierarchies—e.g., the number of objects a givendirectory can store and the number of levels a directory hierarchy maycontain. The ability to seamlessly add nodes to the various file storageservice subsystems, such as the access subsystem, the metadata subsystemand the storage subsystem may help alleviate such scalabilitylimitations. The logical separation of the metadata from the data mayhelp achieve desired distinct levels of performance, availability anddurability for both metadata and data, without imposing the requirementsof the metadata (which may have more stringent needs) on the data. Forexample, metadata may be preferentially stored on SSDs, while data maybe accommodated on less expensive rotating disk-based devices. Otherstorage systems in provider network environments may not support thefamiliar file system interfaces and the consistency semantics of thekinds that many applications are designed to rely on.

The optimistic concurrency control mechanisms described, including theconditional write mechanism for single-page writes and the distributedtransaction scheme for multi-page writes, may help to avoid some of thetypes of bottlenecks that typically arise when more traditionallocking-based schemes are used. Extent oversubscription and variablestripe sizing may be used to manage tradeoffs between space utilizationefficiency and metadata size. The offset-based congestion controltechniques may help improve overall I/O performance for certain types ofapplications, e.g., applications in which a given configuration file mayhave to be read by large numbers of concurrent client threads atapplication startup. The object renaming technique may help ensure filesystem consistency in the event of metadata node failures that mayinevitably arise in large distributed file stores. The namespacemanagement techniques discussed earlier may be used to implement filesystems with millions of objects (even within a single directory) whilemaintaining relatively flat response times as the number of objectsincreases. The client session management caching and lease renewaltechniques may help keep session-related overhead low. The loadbalancing and rebalancing approaches may help to reduce the likelihoodof overload-induced failures.

Illustrative Computer System

In at least some embodiments, a server that implements a portion or allof one or more of the technologies described herein, including thetechniques to implement the components of the access, metadata andstorage subsystems of the distributed file storage service and/or loadbalancer nodes may include a general-purpose computer system thatincludes or is configured to access one or more computer-accessiblemedia. FIG. 69 illustrates such a general-purpose computing device 9000.In the illustrated embodiment, computing device 9000 includes one ormore processors 9010 coupled to a system memory 9020 (which may compriseboth non-volatile and volatile memory modules) via an input/output (I/O)interface 9030. Computing device 9000 further includes a networkinterface 9040 coupled to I/O interface 9030.

In various embodiments, computing device 9000 may be a uniprocessorsystem including one processor 9010, or a multiprocessor systemincluding several processors 9010 (e.g., two, four, eight, or anothersuitable number). Processors 9010 may be any suitable processors capableof executing instructions. For example, in various embodiments,processors 9010 may be general-purpose or embedded processorsimplementing any of a variety of instruction set architectures (ISAs),such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitableISA. In multiprocessor systems, each of processors 9010 may commonly,but not necessarily, implement the same ISA. In some implementations,graphics processing units (GPUs) may be used instead of, or in additionto, conventional processors.

System memory 9020 may be configured to store instructions and dataaccessible by processor(s) 9010. In at least some embodiments, thesystem memory 9020 may comprise both volatile and non-volatile portions;in other embodiments, only volatile memory may be used. In variousembodiments, the volatile portion of system memory 9020 may beimplemented using any suitable memory technology, such as static randomaccess memory (SRAM), synchronous dynamic RAM or any other type ofmemory. For the non-volatile portion of system memory (which maycomprise one or more NVDIMMs, for example), in some embodimentsflash-based memory devices, including NAND-flash devices, may be used.In at least some embodiments, the non-volatile portion of the systemmemory may include a power source, such as a supercapacitor or otherpower storage device (e.g., a battery). In various embodiments,memristor based resistive random access memory (ReRAM),three-dimensional NAND technologies, Ferroelectric RAM, magnetoresistiveRAM (MRAM), or any of various types of phase change memory (PCM) may beused at least for the non-volatile portion of system memory. In theillustrated embodiment, program instructions and data implementing oneor more desired functions, such as those methods, techniques, and datadescribed above, are shown stored within system memory 9020 as code 9025and data 9026.

In one embodiment, I/O interface 9030 may be configured to coordinateI/O traffic between processor 9010, system memory 9020, and anyperipheral devices in the device, including network interface 9040 orother peripheral interfaces such as various types of persistent and/orvolatile storage devices used to store physical replicas of data objectpartitions. In some embodiments, I/O interface 9030 may perform anynecessary protocol, timing or other data transformations to convert datasignals from one component (e.g., system memory 9020) into a formatsuitable for use by another component (e.g., processor 9010). In someembodiments, I/O interface 9030 may include support for devices attachedthrough various types of peripheral buses, such as a variant of thePeripheral Component Interconnect (PCI) bus standard or the UniversalSerial Bus (USB) standard, for example. In some embodiments, thefunction of I/O interface 9030 may be split into two or more separatecomponents, such as a north bridge and a south bridge, for example.Also, in some embodiments some or all of the functionality of I/Ointerface 9030, such as an interface to system memory 9020, may beincorporated directly into processor 9010.

Network interface 9040 may be configured to allow data to be exchangedbetween computing device 9000 and other devices 9060 attached to anetwork or networks 9050, such as other computer systems or devices asillustrated in FIG. 1 through FIG. 68, for example. In variousembodiments, network interface 9040 may support communication via anysuitable wired or wireless general data networks, such as types ofEthernet network, for example. Additionally, network interface 9040 maysupport communication via telecommunications/telephony networks such asanalog voice networks or digital fiber communications networks, viastorage area networks such as Fibre Channel SANs, or via any othersuitable type of network and/or protocol.

In some embodiments, system memory 9020 may be one embodiment of acomputer-accessible medium configured to store program instructions anddata as described above for FIG. 1 through FIG. 68 for implementingembodiments of the corresponding methods and apparatus. However, inother embodiments, program instructions and/or data may be received,sent or stored upon different types of computer-accessible media.Generally speaking, a computer-accessible medium may includenon-transitory storage media or memory media such as magnetic or opticalmedia, e.g., disk or DVD/CD coupled to computing device 9000 via I/Ointerface 9030. A non-transitory computer-accessible storage medium mayalso include any volatile or non-volatile media such as RAM (e.g. SDRAM,DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., that may be included in someembodiments of computing device 9000 as system memory 9020 or anothertype of memory. Further, a computer-accessible medium may includetransmission media or signals such as electrical, electromagnetic, ordigital signals, conveyed via a communication medium such as a networkand/or a wireless link, such as may be implemented via network interface9040. Portions or all of multiple computing devices such as thatillustrated in FIG. 69 may be used to implement the describedfunctionality in various embodiments; for example, software componentsrunning on a variety of different devices and servers may collaborate toprovide the functionality. In some embodiments, portions of thedescribed functionality may be implemented using storage devices,network devices, or special-purpose computer systems, in addition to orinstead of being implemented using general-purpose computer systems. Theterm “computing device”, as used herein, refers to at least all thesetypes of devices, and is not limited to these types of devices.

CONCLUSION

Various embodiments may further include receiving, sending or storinginstructions and/or data implemented in accordance with the foregoingdescription upon a computer-accessible medium. Generally speaking, acomputer-accessible medium may include storage media or memory mediasuch as magnetic or optical media, e.g., disk or DVD/CD-ROM, volatile ornon-volatile media such as RAM (e.g. SDRAM, DDR, RDRAM, SRAM, etc.),ROM, etc., as well as transmission media or signals such as electrical,electromagnetic, or digital signals, conveyed via a communication mediumsuch as network and/or a wireless link.

The various methods as illustrated in the Figures and described hereinrepresent exemplary embodiments of methods. The methods may beimplemented in software, hardware, or a combination thereof. The orderof method may be changed, and various elements may be added, reordered,combined, omitted, modified, etc.

Various modifications and changes may be made as would be obvious to aperson skilled in the art having the benefit of this disclosure. It isintended to embrace all such modifications and changes and, accordingly,the above description to be regarded in an illustrative rather than arestrictive sense.

1.-24. (canceled)
 25. A system, comprising: one or more computingdevices configured to: receive an I/O (input/output) request directed toa portion of a storage object of a storage service, wherein the storageobject comprises a plurality of logical blocks; select a prioritizationpolicy for concurrent requests directed to at least a portion of theplurality of logical blocks; determine, in accordance with the selectedprioritization policy, a congestion control parameter value to be usedto schedule a particular storage operation corresponding to the I/Orequest; and schedule the particular storage operation at a selectedphysical storage location in accordance with the determined congestioncontrol parameter value.
 26. The system as recited in claim 25, whereinthe congestion control parameter value indicates one of: (a) a delayperiod after which the particular storage operation is to be initiatedor (b) a cost assigned to the particular storage operation, wherein thecost is to be used to determine a sequence in which the particularstorage operation is to be initiated relative to other storageoperations corresponding to I/O requests directed to the portion of theplurality of logical blocks.
 27. The system as recited in claim 25,wherein the one or more computing devices are further configured to:select the prioritization policy from among a plurality ofprioritization policies including a first prioritization policy for I/Orequests that include reads, and a second prioritization policy for I/Orequests that include writes.
 28. The system as recited in claim 25,wherein the one or more computing devices are further configured to:select the prioritization policy from among a plurality ofprioritization policies including a first prioritization policy for I/Orequests identified as belonging to a sequential access pattern and asecond prioritization policy for I/O requests identified as randomaccess requests.
 29. The system as recited in claim 25 wherein thestorage service implements one of: (a) a file system interface, (b) aweb services interface in which the storage object is associated with auniversal record identifier (URI), or a block-level device interface.30. A method, comprising: performing, by one or more computing devices:receiving an I/O request directed to a portion of a storage object of astorage service, wherein the storage object comprises a plurality oflogical blocks; determining that the I/O request is directed to at leasta portion of a particular logical block; selecting a prioritizationpolicy from among a plurality of prioritization policies for concurrentrequests directed to a same logical block; computing, in accordance withthe selected prioritization policy, a congestion control parameter valueto be used to schedule a particular storage operation corresponding tothe I/O request; and scheduling the particular storage operation, inaccordance with the computed congestion control parameter value, at aselected physical storage device to which the particular logical blockis mapped.
 31. The method as recited in claim 30, wherein the I/Orequest is directed to a sub-unit of the particular logical blocklocated at a particular offset within the particular logical block,wherein the congestion control parameter value comprises a particularcost assigned to the particular storage operation, relative to costsassigned to other storage operations directed to other offsets withinthe particular logical block, further comprising: scheduling anotherstorage operation, corresponding to a different I/O request directed toa different offset within the particular logical block, before theparticular storage operation in response to a determination that a costassigned to the other storage operation is lower than the particularcost.
 32. The method as recited in claim 31, wherein the different I/Orequest is part of a sequential read pattern of I/O operations requestedby a particular client, wherein the sequential read pattern includes asubsequent I/O request received after the particular storage request,further comprising performing, by the one or more computing devices: inresponse to determining that the other storage operation scheduledcorresponding to the different I/O request has been issued, schedulingone or more subsequent storage operations corresponding to thesubsequent I/O request prior to scheduling the particular storageoperation.
 33. The method as recited in claim 31, further comprising:selecting a function, to which the particular offset within theparticular logical block is provided as input, to be used to generatethe particular cost assigned to the particular storage operation,wherein, for a first offset and a second offset provided as input,wherein if the first offset is larger than the second offset, themathematical function generates a higher cost corresponding to the firstoffset than a cost generated for the second offset.
 34. The method asrecited in claim 30, wherein the I/O request is directed to a sub-unitof the particular logical block, the method further comprisingperforming, by the one or more computing devices: sending, based on theselected prioritization policy, an error response to a client from whomanother I/O request directed to a different sub-unit of the particularlogical block is received; and receiving, after said error message issent, a retry request corresponding to the other I/O request.
 35. Themethod as recited in claim 30, wherein said computing the congestioncontrol parameter value is performed in response to determining that aworkload associated with the particular logical block is above athreshold level.
 36. The method as recited in claim 30, wherein contentsof the particular logical block are stored at a particular storagedevice, the method further comprising performing, by the one or morecomputing devices: changing the threshold level based at least in parton one or more of: (a) an identification of a client from which one ormore I/O requests directed to the particular storage device arereceived, (b) a time of day, (c) a workload metric of the particularstorage device derived from measurements collected over a particulartime period, or (d) a name of a file store object whose contents arelocated at the particular storage device.
 37. The method as recited inclaim 30, wherein the computed congestion control parameter valueindicates a delay period after which the particular storage operation isto be initiated.
 38. The method as recited in claim 37, wherein the I/Orequest is directed to a sub-unit of the particular logical block,wherein the delay period is computed using a function to which an offsetof the sub-unit is provided as input.
 39. The method as recited in claim39, wherein the delay period is computed using a random numbergenerator.
 40. The method as recited in claim 30, further comprisingperforming, by the one or more computing devices: selecting theprioritization policy from among the plurality of prioritizationpolicies including a first prioritization policy for I/O requests thatinclude reads, and a second prioritization policy for I/O requests thatinclude writes.
 41. The method as recited in claim 30, furthercomprising performing, by the one or more computing devices: selectingthe prioritization policy from among the plurality of prioritizationpolicies including a first prioritization policy for I/O requestsidentified as belonging to a sequential access pattern, and a secondprioritization policy for I/O requests identified as random accessrequests.
 42. The method as recited in claim 30, further comprisingperforming, by the one or more computing devices: selecting theprioritization policy from among the plurality of prioritizationpolicies including a first prioritization policy for I/O requestsinvolving a storage operation of a first size range, and a secondprioritization policy for I/O requests involving a storage operation ofa different size range.
 43. The method as recited in claim 30, whereinthe storage service implements one of: (a) a file system interface, (b)a web services interface in which the storage object is associated witha universal record identifier (URI), or (c) a block-level deviceinterface.
 44. A non-transitory computer-accessible storage mediumstoring program instructions that when executed on one or moreprocessors: receive an I/O request directed to a portion of a storageobject managed at a distributed storage service; select a prioritizationpolicy for concurrent requests directed a common portion of physicalstorage of the storage service; determine, based at least in part onselect the selected prioritization policy, a congestion controlparameter value to be used to schedule a particular storage operationcorresponding to the I/O request; and schedule the particular storageoperation, in accordance with the determined congestion controlparameter, at a selected physical storage device to which the portion ofthe storage object is mapped.